Troubleshooting⚓︎

The operator reports everything it knows about a Site on the Site itself. Start every investigation with kubectl describe site, read the conditions and the last reconcile error, and only then move to pods and logs. This page lists what each condition and reason means and where to look next.

First look⚓︎

kubectl -n xcore get sites.expeto.io
kubectl -n xcore describe site <name>

The Status section shows the conditions, Last Reconcile Time (updated on every pass, so a stale value means the operator is not running) and, when the last pass failed, Last Reconcile Error with the wrapped error verbatim. The Events section below it records every condition transition.

Status:
  Conditions:
    Type:     InfrastructureReady   Status: True   Reason: InfrastructureReady
    Type:     GatewaysReady         Status: True   Reason: AllGatewaysReady    Message: 3 child UPF(s) ready
    Type:     ControlPlaneReady     Status: True   Reason: ControlPlaneReady   Message: Control-plane children ready: smf, mm, mme, sdm
    Type:     Ready                 Status: True   Reason: Ready               Message: Site agent and all 3 child UPF(s) are ready
  Last Reconcile Time:  2026-10-26T16:29:11Z

Conditions⚓︎

Condition	Meaning
`Ready`	Rollup of the Agent, every user plane and every control-plane function, and the condition every validation failure is reported on. The one to alert on.
`GatewaysReady`	Every child user plane reports ready.
`ControlPlaneReady`	Every control-plane child the topology needs reports ready: `smf`, `mm` (the AMF), `mme` and `sdm` (the subscriber store). Absent on a Site without them.
`PeeringReady`	The operator can manage BGP resources in MetalLB's namespace. Informational; never part of `Ready`.
`InfrastructureReady`	Always `True`; kept so that existing dashboards and alerts keep working.

The child resources (kubectl -n xcore get smf,upf,mm,mme,sdm) carry their own conditions: NetworkConfigured says whether the pods' network attachments exist, and the mme and sdm resources' DatabaseReady whether their session store is up. kubectl describe on a child that is not ready shows which.

Reasons⚓︎

A False condition names its cause in reason. The catalogue:

Reason	Condition	What it means	What to check
`ProvisioningAgent`, `ProvisioningGateways`, `ProvisioningControlPlane`	`Ready`	Normal start-up: the Site is converging and waits for the Agent, the user planes or the control-plane functions.	Nothing yet. A fresh Site takes a few minutes; the Ignite initialisation job runs first. If it does not settle, the matching subsystem condition and `kubectl get pods` show what is waiting.
`TopologyMissingNADs`	`Ready`	A topology interface references an attachment name that `spec.network.attachments` does not declare.	Compare the `name` under `n2`, `s1`, `n3`, `n6`, `n9` with the attachment names; they are matched exactly.
`TopologyInvalidUpfName`	`Ready`	Duplicate user-plane names or `systemId` values, an anchor without a `systemId`, a front with one, a missing `mode`, or a user-plane name that derives the Site's own service identifier.	Every user plane needs a unique name and a `mode`, and every anchor (a user plane with `n6`) a unique `systemId`; a front (no `n6`) sets none.
`NetworkSpecInvalid`	`Ready`	`spec.network` or the topology addressing failed validation: a bad peer address, a peering or range referenced but not declared, more than one `metallb` BGP router, a `gobgp` router whose first peering's attachment is not an `n6` entry with a `transitIpRange` on a user plane that uses it (`cannot derive localAddr`), an advertisement mixing BGP and layer 2 fields, a transit address that is the subnet's network or broadcast address, or a range that overlaps the cluster's pod or Service network.	The message names the field.
`ControlPlaneSpecInvalid`	`Ready`	`spec.topology.controlPlane` failed validation: no `plmnId` beside a mobility function or subscriber store, an `amf` or `mme` without the subscriber store it authenticates through, AMF and MME identities that map onto each other, an MME TAC above 65535, a mobile reachable timer not longer than T3412, or a duplicate slice.	The message names the field.
`InterfaceRealizationInvalid`	`Ready`	An N2, S1, N3 or N16 entry has neither an attachment nor an ingress address, references an unknown service range, pins an address outside the range, or declares several addresses without `canonicalIp`.	The message names the entry.
`ReachabilityModeConflict`	`Ready`	Pools on one user plane declare different reachability modes.	A user plane supports one mode across all of its pools; split the pools across user planes.
`IgniteConfigInvalid`	`Ready`	Ignite heap, data region or resource settings are inconsistent (minimum above maximum, or a memory limit that cannot hold the heap plus the data region).	Adjust `spec.ignite.jvm`, `dataRegion` and `resources` together.
`DragonflySnapshotInvalid`	`Ready`	The Dragonfly snapshot schedule is not a valid cron expression.	Fix `spec.dragonfly.snapshot.cron`.
`MariaDbOperatorMissing`, `DragonflyOperatorMissing`	`Ready`	The CustomResourceDefinition of a data-backend operator is not installed.	Re-enable the chart's bundled operator or install it. The chart always applies these CustomResourceDefinitions, so on a chart install a missing operator shows instead as MariaDB or Dragonfly resources that never become ready.
`MariaDbStaleStorage`	`Ready`	A MariaDB volume from an earlier database is present and does not match the current topology.	Remove the stale claim only if its data is not needed.
`MariaDbReplicasLocked`	`Ready`	`spec.mariaDb.replicas` would move an existing database between standalone and Galera.	That transition is refused in both directions; see Upgrade, change and uninstall.
`GatewayNotReady`	`GatewaysReady`	A user plane's dataplane StatefulSet or controller Deployment is not ready.	`kubectl get pods -l app.kubernetes.io/component=dataplane`; a `Pending` dataplane pod usually cannot get its Multus attachment on that node (parent interface name, missing CNI plugin binary).
`ControlPlaneNotReady`	`ControlPlaneReady`	A control-plane child is not ready; the message names it.	`kubectl -n xcore describe` on the child the message names, and the pods of that function.
`MetalLBRBACMissing`	`PeeringReady`	MetalLB CustomResourceDefinitions exist but the release was installed without `bgp.enabled=true`.	`helm upgrade ... --set bgp.enabled=true`.
`RejectedExtraSite`	`Ready`	An older Site already exists in this namespace.	One Site per namespace; delete the extra one or install a second release elsewhere.
`DrainingWorkloads`	`Ready`	The Site is being deleted and its workloads are draining.	Wait; if it never completes, see stuck deletion.

Pods⚓︎

kubectl -n xcore get pods
kubectl -n xcore get pods --field-selector=status.phase!=Running,status.phase!=Succeeded

Symptom	Likely cause
Dataplane pod `Pending` or `ContainerCreating` with a Multus event	The attachment's `master` interface does not exist on the scheduled node, or the CNI plugin binary (`macvlan`, `static`) is missing from the node's CNI directory. `kubectl describe pod` shows the CNI error.
`ImagePullBackOff`	Wrong registry credentials, or an air-gapped cluster without `global.imageRegistry` pointing at the mirror.
Ignite pod `Pending`	No node has 11 GiB free for the default request; shrink `spec.ignite.resources` and `jvm` on small clusters.
Ignite initialisation job never completes	The Ignite pods cannot reach each other on the discovery Service, or a previous cluster's work directory is on the volume.
Ignite of a new Site fails with `Cluster names do not match`	The volumes of an earlier Site in the namespace are still there. Deleting that Site recorded a `Warning` event with reason `IgniteStorageRetained` that names the claims (`kubectl -n xcore get events --field-selector reason=IgniteStorageRetained`, while the cluster keeps the event); delete them, as described under Uninstall.
A GoBGP router's session does not come up	Read `kubectl -n xcore logs upf-<user plane>-dataplane-0 -c gobgp` and `-c netcfg`. The peer must accept a session opened by the user plane from its N6 transit address; see GoBGP routers.
Agent pod `Pending` on `FailedMount`	`agent.smfRulesConfigMapRef` names a ConfigMap that does not exist or has no `rules.txt` key.
A Deployment keeps restarting after a Site edit	Restart-required fields rolled it; check `kubectl rollout status` and the container log.

Radios and devices⚓︎

Symptom	Likely cause
An eNodeB's S1 Setup or a gNodeB's NG Setup is rejected	The radio broadcasts a PLMN or a TAC the Site does not serve. Compare the radio's configuration with `plmnId` and the `trackingAreas` of the `mme` or `amf` block. In the CLI, `cd /lte` and `read enodeb`, or `cd /nr` and `read ran`, list the radios that are connected.
Every 4G attach fails although S1 Setup succeeds	A 2-digit MNC: the MME needs a 3-digit MNC to read a home IMSI.
The MME keeps an old PLMN or MME identity after the Site changed it	The MME restores its identity from its session store at start-up. Delete the `mme-cache` pod, wait until it is ready again, then delete the MME pod (`kubectl -n xcore delete pod -l app=mme-mme`); attached devices attach again.
Every 4G attach fails when the default bearer is set up, while 5G sessions work	The 4G path has no front: a user plane in mode `4G` or `Dual` with both `n3` and `n6` would be the SGW-U and the PGW-U of the session, which the SMF refuses. Serve the eNodeBs from a front with `n3` and `n9`.
A device is rejected at authentication	The subscriber-store keys: a transport key that differs from the SIM vendor's, a Keyset Mapping whose HSS Identifier differs from the name the key was loaded under, or, on 5G only, a missing SUCI key or a key identifier that differs from the SIM's. See Load the subscriber-store keys.
A Mobile Asset exists in xControl but not in the subscriber store	In the CLI, `cd /debug/ops` and `read sync-sub-error` list the Agent's provisioning failures; `cd /lte` and `read hss <imsi>`, or `cd /nr` and `read udr <imsi>`, show what the store holds.

Preflight⚓︎

The chart's Helm tests re-run the operator's preflight, which reports each dependency:

helm test xcore -n xcore --filter name=xcore-xfabric-test-preflight --logs

Multus CNI, cert-manager and the netcfg operator are required; the others (whereabouts, MetalLB, FRR-K8s, prometheus-operator, duplicate operators) are warnings with a fix hint.

Admission webhook denials⚓︎

A kubectl edit or kubectl patch on an object the operator created is refused:

admission webhook "vmanaged.expeto.io" denied the request: ... managed by the Site; edit the Site instead

This is intended: change the Site field that renders the object. If the webhook is unreachable (failurePolicy: Fail) because the operator pod is down, restore the operator first.

Uninstall refused⚓︎

helm uninstall fails while a Site exists in the release namespace: the chart's pre-delete hook refuses to remove the operator, deletes nothing, and leaves the release in the uninstalling state. Delete the Sites and run the uninstall again, or let the hook delete them; see If the uninstall is refused.

Stuck deletion⚓︎

A Site whose operator was removed before the Site was deleted stays in Terminating, held by the expeto.io/bgp-cleanup finalizer that only the operator can process. Reinstall the operator and let it finish. Only if that is impossible, and together with Expeto Support, remove the finalizer and delete any BGP resources the Site created in MetalLB's namespace by hand:

kubectl -n xcore patch site <name> --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl -n metallb-system get bgppeers,frrconfigurations,bfdprofiles,ipaddresspools,bgpadvertisements \
    -l expeto.io/site=<name>

Operator logs⚓︎

kubectl -n xcore logs deployment/xcore-xfabric --tail=200

Every reconcile step wraps its error with the step name, so the log line that matches Last Reconcile Error on the Site says which phase failed.