Troubleshooting

This page provides commands and procedures for diagnosing issues with your NetFoundry Self-Hosted environment.

Check deployment status

Run nf-status to get a quick overview of all NetFoundry deployments across the ziti, support, and cert-manager namespaces:

nf-status

All deployments should show the expected replica count in the READY column (e.g., 1/1). If any show 0/1, investigate further using the commands below.

Check pod health

List pods in the ziti namespace:

kubectl get pods -n ziti

List pods in the support namespace:

kubectl get pods -n support

The Beats agents run in their own namespace:

kubectl get pods -n beats

Healthy pods show Running with all containers ready. Common unhealthy states:

Status	Meaning
Pending	Pod can't be scheduled—usually insufficient CPU, memory, or storage
CrashLoopBackOff	Container is crashing repeatedly—check logs for the root cause
ImagePullBackOff	Can't pull the container image—check registry credentials and network access
Error	Container exited with an error—check logs

To get details on why a pod is unhealthy:

kubectl describe pod <pod-name> -n ziti

Check Kubernetes events

Events reveal scheduling failures, volume issues, and other cluster-level problems. View recent events sorted by time:

kubectl get events -n ziti --sort-by='.metadata.creationTimestamp'
kubectl get events -n support --sort-by='.metadata.creationTimestamp'

To watch events in real time:

kubectl get events --watch -n ziti

Check services and external access

Verify that LoadBalancer services have been assigned external addresses:

kubectl get services -n ziti

If the EXTERNAL-IP column shows <pending>, the cluster's load balancer provisioner may not be configured or may be unable to allocate an address.

View logs

Controller logs

kubectl logs -f deployment/ziti-controller -n ziti

Edge router logs

kubectl logs -f deployment/ziti-router-1 -n ziti

Support stack component logs

kubectl logs -f deployment/grafana -n support
kubectl logs -f deployment/nf-data-connector -n support

Beats agent logs

kubectl logs -f daemonset/filebeat -n beats
kubectl logs -f daemonset/metricbeat -n beats

View previous container logs (after a crash)

If a container has restarted, view the logs from the previous instance:

kubectl logs <pod-name> -n ziti --previous

Restart a component

If a component is in a bad state after ruling out configuration issues, restart its deployment:

kubectl rollout restart deployment ziti-controller -n ziti
kubectl rollout restart deployment ziti-router-1 -n ziti

For support stack components:

kubectl rollout restart deployment grafana -n support
kubectl rollout restart deployment nf-data-connector -n support

No OpenZiti data in Grafana or Kibana

All OpenZiti events and metrics are collected by nf-data-connector. If the Grafana ziti.* dashboards are empty, start there.

1. Confirm the connector is running:

   kubectl get pods -n support -l app=nf-data-connector

   If the command returns no pods, check the replica count:

   kubectl get deployment nf-data-connector -n support

   A READY of 0/0 means the connector was rendered with replicas: 0 because dataConnector.controllerAddress in support-values.yml is still the default localhost:1280. Set it to your controller's client API address and port, then re-run the support helm upgrade.

2. Check that the connector has OpenZiti credentials:

   kubectl get secret data-connector-credentials -n support

   This secret is created by the data-connector-create-ziti-admin hook job during install and upgrade. If it is missing, inspect the job's logs:

   kubectl logs job/data-connector-create-ziti-admin -n support

   The job skips itself (without failing) when the controller is not yet reachable. Re-running the support helm upgrade fires it again.

3. Check the connector's own logs for controller or Elasticsearch connection errors:

   kubectl logs -f deployment/nf-data-connector -n support

4. If OpenZiti data is present but container logs are missing, the problem is on the Filebeat path instead. Filebeat writes to the ziti.logs-v2 index and needs the Elasticsearch credentials and CA replicated into the beats namespace:

   kubectl get pods -n beats
   kubectl get secret elasticsearch-es-elastic-user elasticsearch-es-http-certs-public -n beats

   On an external-Elasticsearch install, look for external-es-creds (and external-es-ca if you supplied a CA) instead. Pods stuck in ContainerCreating with a secret not found event mean those secrets were not replicated; re-running nf-upgrade --support replicates them.

Verify controller connectivity

Test that the controller API is reachable at its advertise address:

curl --insecure https://<controller-advertise-address>:<port>/version

If this fails, check DNS resolution and firewall rules:

nslookup <controller-advertise-address>

Log into the controller CLI

Use nf-login to authenticate with the controller and run Ziti CLI commands:

nf-login

Once logged in, you can inspect network state directly:

ziti edge list identities
ziti edge list services
ziti edge list edge-routers

Access the controller pod directly

For advanced debugging, you can exec into the controller pod:

kubectl exec -it deployment/ziti-controller -n ziti -- /bin/bash

Collect diagnostics for NetFoundry support

Run the support bundle command to collect logs and stack dumps into a zip file:

nf-support-bundle

This collects the last 24 hours of logs from all pods in the ziti and support namespaces, plus ziti agent stack output from each Ziti pod. Beats logs are not included — collect them separately with kubectl logs daemonset/filebeat -n beats if container-log ingestion is the problem.

For additional diagnostics, pass optional flags:

nf-support-bundle --mem    # Include memory statistics
nf-support-bundle --cpu    # Include CPU profiling
nf-support-bundle --heap   # Include heap profiling

Include the generated support_bundle_*.zip file along with the log files in the logs/ directory when contacting NetFoundry support.