Upgrade, Change and Uninstall⚓︎

Once a Site is Ready, day-two work falls into three kinds: upgrading the Helm release (the operator, the Agent and the bundled operators), changing the Site definition, and removing a core. This page covers each, plus the topology decisions that are fixed at creation and the customer-owned overrides the operator supports.

Upgrade the release⚓︎

helm repo update
helm upgrade xcore expeto/xfabric -n xcore -f values.yaml --server-side --force-conflicts \
    --set global.imageCredentials.username='<username>' \
    --set global.imageCredentials.password='<password>' \
    --set agent.apiToken='<site API token>'

The values file is yours: it holds every setting of the installation, lives in version control, and is passed on every upgrade together with the secrets, exactly as at install time. Anything it does not mention takes the chart's default for the new release. If no values file is kept, --reuse-values carries the current release's values forward, secrets included:

helm upgrade xcore expeto/xfabric -n xcore --reuse-values --server-side --force-conflicts

Use --server-side --force-conflicts on every upgrade. Some dependency CustomResourceDefinitions exceed the size that client-side apply can annotate, and server-side apply lets Helm take ownership of fields another manager (a kubectl apply of the CRDs, for instance) touched earlier. The flags are safe on helm install too.

The chart applies the release's CustomResourceDefinitions before anything else in the upgrade, so the Site schema of the new release is in place when the new operator starts; an older release never downgrades them. The upgrade then rolls the operator, which reconciles the Site with the new network-function images. Restart-required changes roll the affected pods one workload at a time; the data backends keep their volumes.

Upgrade xControl first: the Agent that an xCore release pins must not be newer than the xControl it registers with. A release whose Site schema changes lists the edits an existing Site needs; plan them with Expeto Support before the upgrade window.

Upgrade from 4.0.5 to 4.0.6⚓︎

xCore 4.0.6 needs xControl 3.3.22 or later. Its Site schema changes, so review the Site against the list below with Expeto Support, and apply the edited Site right after helm upgrade, in the same window. Until it is applied, the operator reports the Site Ready=False with a validation reason and does not reconcile it.

• Front user planes lose systemId. A user plane without n6 (an SGW-U or an intermediate UPF) serves every System and must not set systemId; remove it, typically from the SGW-U of a public core. Anchors keep theirs.
• subscriberManagement.plmnId becomes controlPlane.plmnId, with the same value. It is required whenever amf, mme or the subscriber store is declared.
• controlPlane.mm becomes controlPlane.amf: rename the block, rename ranInterfaces to n2 and remove mode.
• amf.exposure is gone. exposure.loadBalancer: {range, ip} becomes n2: [{ingress: [{range, ip}]}].
• amf.servedTrackingAreas becomes trackingAreas, a list of integers: [{tac: "000001"}] becomes [1]. Tracking area 0 is refused.
• amf.allowedRanNodeIds is removed; nothing ever read it.
• amf.n2 and mme.s1 take one entry. Keep the first and remove the rest.
• amf and mme need their subscriber store: subscriberManagement enabled in mode 5G or Dual for the AMF, 4G or Dual for the MME.
• subscriberManagement.adminPass, adminPassSecret and dbInterfaces are removed, and spec.hickory is removed. Delete the keys.
• Control-plane validation failures report ControlPlaneSpecInvalid instead of NetworkSpecInvalid. Update alerts that match on the reason.
• The SMF collects session usage every 59 minutes instead of every minute, and a user plane still reports every 10 MiB. Session usage in xControl therefore updates less often for light sessions. controlPlane.usageReporting sets both values.
• A Site with a 5G subscriber store provisioned under 4.0.5 needs its UDR records migrated to the new UDR's identifier format before 5G devices can register again; Expeto Support runs the migration.

Change a Site⚓︎

Apply a change by editing the Site definition in version control and applying it again. The Site resource reference says how each field behaves; they fall into two classes:

To enforce a change window, set spec.maintenanceMode: false on the Site. The admission webhook then rejects any edit that would require a restart until you set it back to true; live-reload edits still go through. Toggling the flag itself is never gated.

Two settings are fixed once created and cannot be changed by editing the Site:

• User-plane session table: numberOfSessions on a user plane is immutable after the user plane exists. Size it for the peak; the dataplane rejects new sessions with PFCP cause 75 when the table is full.
• MariaDB topology: see below.

namePrefix renames every resource on the next reconcile but deletes none of the old ones; treat it as set at install time.

A changed plmnId or mme.identity reaches the MME only once its session store is recreated: the MME restores its PLMN and identity from the store at start-up. After applying such a change, delete the mme-cache pod, wait until it is ready again, then delete the MME pod (kubectl -n xcore delete pod -l app=mme-mme). Attached 4G devices attach again.

MariaDB topology⚓︎

spec.mariaDb.replicas: 1 runs a standalone MariaDB; 3 or another odd number runs a Galera cluster managed by the MariaDB operator, with one replica per node and automatic primary failover. The SMF and the subscriber store always write through the primary Service.

Moving an existing database between the two shapes is refused in both directions with the reason MariaDbReplicasLocked: the operator relocates the data directory when converting to Galera and would not carry the data over, and a cluster bootstrapped down to one node may pick a stale replica. Choose the topology at first install. Growing or shrinking a Galera cluster between odd sizes (3 to 5) is supported; the new members state-transfer from the existing ones.

For a Galera cluster:

• one schedulable node per replica, or set spec.mariaDb.antiAffinity: false on test clusters (colocated replicas defeat the purpose);
• one data volume and one small configuration volume per replica;
• a PodDisruptionBudget keeps replicas - 1 pods available, so node drains proceed one at a time.

Ignite sizing⚓︎

Each Ignite replica requests and is limited to 11 GiB of memory by default, with a 25 GiB work volume, a 2 GiB to 4 GiB heap and a 1 GiB data region. The default engine (aipersist) keeps the dataset on the volume and survives restarts. On small clusters shrink resources, jvm and dataRegion together; the operator refuses a combination whose memory limit cannot hold the heap plus the data region.

Custom SMF rules⚓︎

The SMF routes sessions with a bundled rule set: it selects the Site's SGW-U and intermediate UPF by type, and the anchor by the System of the Mobile Asset (systemId). That covers public, private and combined cores with one front. A Site with more than one SGW-U or intermediate UPF, Mobile Assets provisioned without a System, or special routing (an external P-GW, APN-based steering) needs its own Drools rules instead:

apiVersion: v1
kind: ConfigMap
metadata:
  name: acmecorp-smf-rules
  namespace: xcore
data:
  rules.txt: |
    package com.expeto.smf;
    // your rules

helm upgrade xcore expeto/xfabric -n xcore --reuse-values --set agent.smfRulesConfigMapRef=acmecorp-smf-rules

The ConfigMap is yours: the operator only mounts it. It must live in the install namespace with a rules.txt key, or the Agent pod stays Pending on FailedMount. Nothing validates the rules before the Agent's rule engine loads them at start-up; a bad rule set shows as an Agent crash loop. The Agent reads the file once, so restart it after editing:

kubectl -n xcore rollout restart statefulset/xcore-xfabric-agent

BGP⚓︎

A Site announces UE pools and service addresses through the BGP routers in spec.network.bgp. Each router's impl says where it runs: metallb, the default, or gobgp. The fields are in the Site resource reference.

MetalLB routers⚓︎

A metallb router runs on FRR-K8s. The operator emits the BGP resources into MetalLB's namespace, which needs cross-namespace permissions the chart provisions only when asked:

helm upgrade xcore expeto/xfabric -n xcore --reuse-values --set bgp.enabled=true

MetalLB must already be installed (the chart can bundle it with metallb.enabled=true on clusters that have none). Its namespace is detected; set metallbNamespace when detection fails. Emitted resources carry the labels expeto.io/site and expeto.io/site-namespace, and a finalizer on the Site removes them on deletion. A Site has at most one metallb router, and only that router announces service ranges.

GoBGP routers⚓︎

A gobgp router runs as a GoBGP sidecar in the dataplane pod of each user plane whose UE pools name one of its peerings. The user plane's netcfg sidecar configures it, so it needs neither MetalLB nor bgp.enabled. Agree its use with Expeto Support. In this example, user plane anchor announces 10.45.0.0/16 to the border router at 192.0.2.1:

spec:
  network:
    bgp:
      routers:
        - name: upf-uplink
          asn: 64512
          impl: gobgp
          peerings:
            - name: n6-upstream
              attachment: n6-internet
              peers:
                - name: border
                  address: 192.0.2.1
                  peerAsn: 64500
  topology:
    userPlanes:
      - name: anchor
        mode: Dual
        systemId: 2
        n6:
          - name: n6-internet
            transitIpRange: 192.0.2.10/24
            dataNetworkName: internet
            ueIpPools:
              - cidr: 10.45.0.0/16
                reachability:
                  bgp:
                    peering: n6-upstream

The sidecar announces the pool with 192.0.2.10 as next hop, and uses it as router-id unless the router sets routerId. Plan a GoBGP router within these constraints:

• Local address. The user plane's address on the router is the transitIpRange address of its n6 entry whose name is the attachment of the router's first peering. Without such an entry the Site goes Ready=False with reason NetworkSpecInvalid and a message containing cannot derive localAddr.
• Sessions. The user plane opens every session and accepts none, so the peer must accept a session opened from the user plane. The peer must be directly reachable on that N6 segment; there is no multihop.
• Announcements. The router announces IPv4 UE pools only. Routes the peer sends are not installed. Service ranges on a GoBGP peering are accepted but never announced; put them on the metallb router.
• Peer fields. Only address, peerAsn, holdTime, keepaliveTime and gracefulRestart are used; the other peer fields are ignored.
• One router per user plane. When a user plane's pools name the peerings of two GoBGP routers, every pool is announced through the first one. Keep each user plane on one GoBGP router.
• One replica. Every replica of the user plane announces from the same address, so run a user plane with a GoBGP router at replicas: 1.
• Restarts. Giving a user plane its first GoBGP pool, or removing its last, restarts its dataplane pod, even outside maintenance mode; make that change in a maintenance window. Other changes, to peers or pools, are applied live.
• Control API. GoBGP's API listens without authentication on TCP port 50051 on every interface of the dataplane pod. Restrict that port on the pod network with a NetworkPolicy. A NetworkPolicy does not cover the N6 attachment, so keep untrusted hosts off that segment.

Each sidecar logs to its own container:

kubectl -n xcore logs upf-anchor-dataplane-0 -c gobgp
kubectl -n xcore logs upf-anchor-dataplane-0 -c netcfg

Pin images⚓︎

A release normally runs the images its chart version names. To pin the operator, the Agent or a single network function:

image:
  tag: "<operator build>"
agent:
  image:
    tag: "<agent build>"
imageOverrides:
  oak_smf: docker.expeto.io/oak/smf:<build>

The override keys are listed in the chart's files/operator/versions.yaml (helm show values expeto/xfabric prints them). The operator replaces an override's registry host with global.imageRegistry, so the image must be in that registry. The Agent must never be newer than the xControl it registers with; xControl stays compatible with older Agents, not the reverse. An Agent older than 3.3.22-beta8 cannot manage the MME, so only a Site without an MME may pin an older Agent, and no older than 3.3.20.

Uninstall⚓︎

Delete the Site first, then the release. The operator must be running to remove what it created in other namespaces (the BGP resources) and to let the Site's finalizers complete; an uninstall with Sites present is refused by the chart's pre-delete hook.

kubectl -n xcore delete sites.expeto.io --all
kubectl -n xcore wait --for=delete sites.expeto.io --all --timeout=10m
helm uninstall xcore -n xcore

When the uninstall goes ahead, the hook also deletes the operator's taiga-gateways and taiga-nf-config ConfigMaps and, when the chart created it, the image pull Secret.

If the uninstall is refused⚓︎

With a Site present, the pre-delete hook fails and helm uninstall fails with it. Nothing is deleted, but the release stays in the uninstalling state (helm list -n xcore -a), in which helm upgrade is refused. The hook's log lists the Sites and the ways out:

kubectl -n xcore logs job/xcore-xfabric-pre-delete-drain

• Delete the Sites, then uninstall again. This is the normal path:

  kubectl -n xcore delete sites.expeto.io --all --wait
  helm uninstall xcore -n xcore
  

• Delete the Sites and the release in one step. Create the hook's mode ConfigMap, then uninstall again. The hook deletes every Site, waits up to preDeleteDrain.timeoutSeconds (180 seconds by default) for them to go, and lets the uninstall proceed:

  kubectl -n xcore create configmap xcore-xfabric-pre-delete-mode --from-literal=mode=drain
  helm uninstall xcore -n xcore
  

  The ConfigMap is named <release>-xfabric-pre-delete-mode, or <release>-pre-delete-mode when the release name contains xfabric. The hook deletes it once the uninstall proceeds.

• Skip the hook with helm uninstall xcore -n xcore --no-hooks. Avoid this: the Sites keep running without an operator, and deleting one later hangs on its finalizer (see stuck deletion).

The chart value preDeleteDrain.mode sets the hook's behaviour for every uninstall: abort, the default, refuses as described; drain always deletes the Sites first and suits test clusters only; disabled installs no hook.

Volumes and CustomResourceDefinitions⚓︎

Helm does not delete the PersistentVolumeClaims of MariaDB, Ignite and the Agent, nor the CustomResourceDefinitions, which other releases on the cluster may use. A leftover Ignite volume stops the Ignite cluster of a new Site in the same namespace from starting, so remove the claims when the data is no longer needed:

kubectl -n xcore get pvc
kubectl -n xcore delete pvc --all
kubectl delete namespace xcore

To remove the CustomResourceDefinitions after the last release on the cluster is gone:

helm pull expeto/xfabric --untar
kubectl delete -f xfabric/charts/xfabric-crds/files/

If a Site is stuck in Terminating because the operator was removed first, see Troubleshooting.