Support stack overview

The NetFoundry support stack gives you observability and troubleshooting capabilities by collecting telemetry and events from across your network. It deploys from the helm-charts/support chart that ships with the installer, and pulls one private image (nf-data-connector) from the NetFoundry container registry. Components are pre-configured to work together, providing telemetry dashboards and searchable logs out of the box.

Changed in 2.0.0

Logstash, RabbitMQ, the Ziti Metrics Processor (ZMP), and Redis have been removed. nf-data-connector is now the single telemetry path on every install, and the Beats agents write straight to Elasticsearch. Upgrading from an earlier release runs a guided migration — see Upgrade your installation.

Stack components

• Elasticsearch: Stores and indexes telemetry and log data, and exposes an API for querying it.
• nf-data-connector: The single telemetry path for OpenZiti events and metrics. It subscribes to the controller's management event stream over a WebSocket, enriches events with identity and service context from an in-memory cache, and writes ziti.* data streams directly to Elasticsearch. It authenticates with a dedicated, least-privilege OpenZiti account (data-connector) created automatically during install. Requires NetFoundry container registry credentials.
• Kibana: Web UI for searching and browsing raw log and telemetry data in Elasticsearch. Ships pre-configured with organized data sources.
• Grafana: Industry-standard dashboard tool for viewing and analyzing metrics. Ships pre-configured with all data sources and the dashboards used internally at NetFoundry.
• Ziti Console Enterprise: Web console for managing your OpenZiti network. Installed automatically with the support stack. Requires NetFoundry container registry credentials (REGISTRY_USERNAME and REGISTRY_PASSWORD).
• Beats agents: Micro-containers deployed as a DaemonSet on all Kubernetes nodes, in the beats namespace. They collect logs (Filebeat) and metrics (Metricbeat) from all pods in the ziti and support namespaces and ship them directly to Elasticsearch.
• OpenZiti tunnelers (ziti-edge-tunnel, optional): By default, none of the support tools are exposed externally. You control how they're exposed, but we recommend accessing them over the OpenZiti network using an OpenZiti tunneler. This eliminates extra open ports and satisfies most compliance requirements.

Resource requirements

The support stack requires a minimum of 4 CPU cores available in the cluster. The installer checks available cores before installation and skips the support stack if this threshold is not met. All other OpenZiti components still install normally.

Storage and index lifecycle defaults vary based on deployment type:

Setting	K3s (single-node)	Multi-node cluster
Elasticsearch storage	30Gi	100Gi
Index rollover size	1GB	3GB
Index rollover age	1d	3d
Max index age	7d	7d

Data flow

Telemetry and event data reaches Elasticsearch over three independent paths:

1. OpenZiti events and metrics: nf-data-connector opens a WebSocket to the controller's management event stream, subscribes to the event types configured in support-values.yml, enriches them, and writes ziti.* data streams (ziti.circuit, ziti.usage, ziti.metrics, ziti.link, ziti.session, and so on) directly to Elasticsearch. The connector pulls from the controller rather than the controller pushing to a broker, so there is no message queue to operate.
2. Host and pod telemetry: Metricbeat scrapes node, pod, container, event, and volume metrics from the ziti and support namespaces and writes metricbeat-* indices directly to Elasticsearch.
3. Container logs: Filebeat harvests container logs from the ziti and support namespaces, decodes the JSON log lines, and writes controller and router logs to the ziti.logs-v2 index directly in Elasticsearch.

Grafana and Kibana then read from Elasticsearch to provide dashboards and searchable log views.

Because Filebeat and Metricbeat write to Elasticsearch directly, the Elasticsearch cluster must have a node holding the ingest role. The installer's generated values include it by default (elasticsearch.node_roles); do not remove it, or the writes are rejected and ziti.logs-v2 is never created.

zLAN installs

On a zLAN install the connector additionally opens a Beats (Lumberjack) input on port 5000 for the zLAN firewall (zfw) sensor. The OpenZiti service that fronts it is named logstash.zfw and intercepts on port 5000, so existing zfw sensors need no reconfiguration. It binds to nf-data-connector.support.svc.cluster.local:5000.

Configuration

The installer generates a support-values.yml file with default settings based on your deployment type (K3s or production). If the file already exists, it is left unchanged. Key settings:

elasticsearch:
  node_count: 1
  node_storage: 100Gi        # 30Gi for K3s
  max_index_age: 7d
  rollover_index_age: 3d     # 1d for K3s
  rollover_index_size: 3GB   # 1GB for K3s

dataConnector:
  controllerAddress: "my-controller.example.com:1280"
  image:
    pullSecretName: netfoundry-pull-secret

• node_count: Number of Elasticsearch nodes. Increase for production high-availability.
• node_storage: Persistent volume size per Elasticsearch node.
• max_index_age: How long an index is retained after it rolls over before the ILM policy deletes it. This is the primary data-retention control.
• rollover_index_age / rollover_index_size: Triggers for creating a new index when the current one reaches the specified age or size.
• dataConnector.controllerAddress: Address and port of the OpenZiti controller client API. The installer sets this from CTRL_ADDR and CLIENT_PORT.
• dataConnector.controllerHosts: Optional list of controller addresses for an HA controller cluster. When set, it takes precedence over controllerAddress.
• dataConnector.replicas: Number of connector instances (default 1).
• dataConnector.enabled: Set to false to install the support stack with no OpenZiti telemetry at all. Beats, Grafana, and Kibana are unaffected.
• dataConnector.subscriptions: The controller event types the connector subscribes to. See helm-charts/support/values.yaml for the shipped list.

warning

If dataConnector.controllerAddress is left at the chart default localhost:1280, the connector renders with replicas: 0 and the account-creation job is skipped, so no OpenZiti telemetry is collected. This is what you get if the support stack is installed without NetFoundry container registry credentials. Set a real controller address and re-run the helm upgrade below to recover.

Adjusting data retention

max_index_age sets the same delete threshold for every ILM policy (controller logs/metrics and ZFW events). To retain, for example, 30 days of data:

elasticsearch:
  max_index_age: 30d

note

max_index_age is measured from the moment an index rolls over, not from when it was created. Effective retention is therefore roughly max_index_age plus the rollover window (rollover_index_age / rollover_index_size), so the oldest data on disk can be slightly older than the configured value.

:::warning Plan storage before increasing retention Disk usage scales roughly linearly with the retention window — 30 days holds about 4× the data of the 7-day default. Increase node_storage to match, and keep rollover_index_size at roughly 1/10th of node_storage so indices roll over and age out smoothly. This matters most on fixed-disk appliances, where Elasticsearch stops indexing if the volume fills. :::

To apply configuration changes after installation:

helm upgrade --install support ./helm-charts/support/ --values support-values.yml -n support

External Elasticsearch

If you already have an Elasticsearch cluster, you can connect the support stack to it instead of deploying one via ECK. See Use external Elasticsearch for installation and configuration instructions.

Forwarding telemetry to other systems

The support chart writes to Elasticsearch only; Grafana and Kibana read from Elasticsearch. If you need telemetry copied into another system, contact NetFoundry support.

Access support tools

By default, none of the support tools are exposed externally. The installer creates OpenZiti services for each tool, making them accessible over the OpenZiti network at the following intercept addresses:

Tool	Intercept address	Port
Grafana	grafana.ziti	80 (HTTP)
Kibana	kibana.ziti	443 (HTTPS)
Elasticsearch	elasticsearch.ziti	443 (HTTPS)

The installer generates a support-user.jwt enrollment token for client access. To connect:

1. Enroll the support-user.jwt token with an OpenZiti client (Desktop Edge, mobile tunneler, or CLI tunneler).
2. Once connected, access the tools at the intercept addresses above (e.g., http://grafana.ziti in your browser).

This approach eliminates the need for any externally exposed ports and satisfies most compliance requirements.

If you need to reach the support tools directly instead of over Ziti, see Expose support tools externally.

Default credentials

Tool	Username	Password
Grafana	admin	admin (you'll be prompted to change this on first login)
Elasticsearch / Kibana	elastic	Auto-generated during installation

To retrieve the Elasticsearch / Kibana password:

kubectl get secrets "elasticsearch-es-elastic-user" -n support \
  -o go-template='{{index .data "elastic" | base64decode}}'

Related tools

• Database snapshots: The quickstart automatically deploys a scheduled job that captures the OpenZiti controller database daily. See Back up your installation for configuration and restore instructions.
• Support bundle: Run nf-support-bundle to collect recent logs and stack dumps from the controller and router into a zip file for sending to NetFoundry support. See Collect diagnostics for NetFoundry support for usage and optional flags.