Operations and Maintenance⚓︎

Alarms, metrics, diagnostics and maintenance procedures an MNO relies on to run the platform: faults are alarmed and cleared, metrics agree with the platform's own state, a support case can be documented completely, and time is consistent across the platform.

ID	Test	Category	Applies to
EXP-OAM-001	Alarm raised and cleared for an induced fault	Operations	4G, 5G · Local radio · Single node, HA cluster, Geo-redundant
EXP-OAM-002	Metrics for capacity and usage	Operations	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
EXP-OAM-003	Diagnostic collection for a support case	Operations	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
EXP-OAM-004	xControl backup and restore	Operations	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
EXP-OAM-005	API token expiry warning	Operations	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
EXP-OAM-006	Time synchronization and timestamp consistency	Operations	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant

EXP-OAM-001 — Alarm raised and cleared for an induced fault⚓︎

Attribute	Value
Status	Accepted
Category	Operations
Components	xCore, xView
Applies to	4G, 5G · Local radio · Single node, HA cluster, Geo-redundant
Duration	About 45 min
Prerequisite procedures	EXP-INS-004 — Monitoring path

Objective. An induced fault raises the corresponding alert in xView within the evaluation window, with the Site and Customer labels, is delivered to the configured notification channel, and clears once the fault is removed.

Service impact

This procedure interrupts service. Run it in a maintenance window or in a pre-production environment.

Preconditions

Platform
- A notification channel (for example e-mail or a webhook) is configured for xView alerts, or the alert view alone is used.
Tooling and access
- Access to the xView alert view.
- Log in to xControl with an account that holds the ADMIN role.
Procedures executed first
- EXP-INS-004 — Monitoring path

Procedure

#	Where	Action	Expected result
1	Infrastructure	Power off the private radio, or disconnect its transport. Note the time.	The radio is unreachable.
2	Tester	Watch the alert view.	The private-radio-connection alert appears with the Site and Customer labels.
3	Tester	Check the notification channel.	The notification for the alert arrived.
4	Infrastructure	Power the radio on again. Note the time.	The radio reconnects (EXP-INS-003).
5	Tester	Watch the alert view.	The alert clears; a resolved notification arrives where the channel supports it.
6	Infrastructure	Repeat with a second fault. Block the xCore's outbound path to xControl for ten minutes, then restore it.	The agent-out-of-contact alert fires and clears in the same way.

Pass criteria

Each alert fires within six minutes of the fault, with correct labels, and is delivered.
Each alert clears within five minutes of the fault being removed.

Evidence collected: xControl screenshot.

EXP-OAM-002 — Metrics for capacity and usage⚓︎

Attribute	Value
Status	Accepted
Category	Operations
Components	xCore, xView
Applies to	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
Duration	About 30 min
Prerequisite procedures	EXP-SES-007 — Multiple UE attach EXP-REC-002 — Usage accounting accuracy

Objective. The connected-device and traffic metrics of a Site agree with xControl and with the traffic actually passed, carry the Site's labels, and are retained.

Preconditions

Tooling and access
- Access to the xView Prometheus query interface.
- Log in to xControl with an account that can view the Mobile Asset under test.
Procedures executed first
- EXP-SES-007 — Multiple UE attach
- EXP-REC-002 — Usage accounting accuracy

Procedure

#	Where	Action	Expected result
1	—	Execute EXP-SES-007 — Multiple UE attach as a prerequisite procedure.	As defined in EXP-SES-007.
2	Tester	Query `xcore_clients{site="<site name>"}`.	The value equals the number of Mobile Assets shown as Connected for the Site in xControl.
3	Tester	Note `xcore_download_bytes_total{site="<site name>"}` and `xcore_upload_bytes_total{site="<site name>"}`, then run the transfers of EXP-REC-002 and query again.	Each counter increased by exactly the IP-layer bytes transferred.
4	Tester	Query the same series over the past 24 hours.	Data points exist across the whole period; nothing is missing.
5	Tester	Inspect the labels of the series.	`customer`, `customerId`, `site` and `siteId` are present and correct.

Pass criteria

The connected-device metric equals xControl's count.
The byte counters increase by exactly the transferred IP-layer bytes, as in EXP-REC-002.
The series are labelled with the Site and Customer and retained for at least 24 hours.

Evidence collected: xControl screenshot.

EXP-OAM-003 — Diagnostic collection for a support case⚓︎

Attribute	Value
Status	Accepted
Category	Operations
Components	xCore, xControl
Applies to	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
Duration	About 45 min
Prerequisite procedures	EXP-SES-001 — UE-initiated attach EXP-SES-002 — UE-initiated detach

Objective. An operations engineer can collect, for one Mobile Asset and one time window, every artifact Expeto support needs, and the artifacts correlate with one another by IMSI and time.

Preconditions

Tooling and access
- Access to the cluster with kubectl, the CLI, xControl and a capture point.
Procedures executed first
- EXP-SES-001 — UE-initiated attach
- EXP-SES-002 — UE-initiated detach

Procedure

#	Where	Action	Expected result
1	—	Execute EXP-SES-001 — UE-initiated attach as a prerequisite procedure.	As defined in EXP-SES-001.
2	—	Execute EXP-SES-002 — UE-initiated detach as a prerequisite procedure.	As defined in EXP-SES-002.
3	Kubernetes	Collect the logs of every core pod for the window of the two procedures (`kubectl logs --since-time=... -n <namespace> <pod>`), and `kubectl get pods -o wide` and `kubectl describe pod` output.	The files are saved with the naming convention in Test environment and tooling.
4	CLI	Run and save `read hss <imsi>` (or `read udr <imsi>`), `read mme -f imsi=<imsi>` (or `read amf <imsi>`) and `read pgw -f imsi=<imsi>` (or `read smf -f imsi=<imsi>`).	The output is saved.
5	xControl	Export the Mobile Asset's Event Log (screenshot or report) and note the Site's Agent Version.	The export is saved.
6	Capture	Save the signalling capture of the window.	The capture is saved.
7	Tester	Correlate the artifacts: find the attach in the capture, the same event in the MME or AMF log and in the xControl Event Log.	The IMSI appears in each artifact and the timestamps agree within one second, as in EXP-OAM-006.

Pass criteria

Every artifact in the list is collected.
The same event is identifiable in the capture, the logs and xControl with consistent timestamps.

Evidence collected: CLI output; xControl screenshot; 3GPP signalling capture (pcap).

EXP-OAM-004 — xControl backup and restore⚓︎

Attribute	Value
Status	Accepted
Category	Operations
Components	xControl, xCore
Applies to	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
Duration	About 2 h
Prerequisite procedures	EXP-INS-002 — Management path and subscriber synchronization EXP-SES-001 — UE-initiated attach

Objective. A backup of the xControl database is restored onto a rebuilt xControl instance. The provisioning state and the accounts are those of the backup, every Site reconnects, and the synchronization check shows exactly the differences that arose after the backup, which a re-synchronization resolves.

Service impact

This procedure interrupts service. Run it in a maintenance window or in a pre-production environment.

Preconditions

Platform
- The backup and restore procedure of the xControl installation guide is at hand, and you can run pg_dump against the xControl database (PostgreSQL 17) and rebuild the xControl instance from the chart.
- A maintenance window is agreed; xControl is unavailable during the restore.
Mobile Asset
- At least one Mobile Asset is provisioned on the Site.
Tooling and access
- Log in to xControl with an account that holds the ADMIN role.
- Access to the cluster with kubectl and to the CLI.
Procedures executed first
- EXP-INS-002 — Management path and subscriber synchronization
- EXP-SES-001 — UE-initiated attach

Procedure

#	Where	Action	Expected result
1	xControl	Record the number of Customers, Sites, Systems, Mobile Assets and Accounts.	The baseline is recorded.
2	Infrastructure	Take the backup with the documented procedure (`pg_dump` of the xControl database, plus the configuration the guide names). Note the time.	The backup completes and its file is stored off the cluster.
3	xControl	Create a new Mobile Asset X (EXP-PRV-001, first step) and a new Account Y.	Both exist in xControl; X reaches the Site (`read hss <imsi>`).
4	Infrastructure	Rebuild the xControl instance from the chart and restore the backup with the documented procedure. Note the time.	xControl starts and the login page is served.
5	xControl	Log in with an account from the baseline and compare the counts with the baseline.	The counts equal the baseline; Mobile Asset X and Account Y do not exist.
6	xControl	Open Sites.	Every Site returns to online within five minutes of xControl starting.
7	CLI	Run `cd /debug/ops`, then `cmd cache-clear type=SUBSCRIBER_CHECK_ERRORS`, `cmd sync-sub-check mode=FULL` (repeat until it no longer answers that a check is already running), then `read sync-sub-error`.	Exactly one discrepancy is reported, Mobile Asset X present on the Site but unknown to xControl.
8	CLI	Run `cmd sync-sub siteWide=true type=RECREATE`, wait for it to finish, then repeat the check.	The error table is empty; `read hss <imsi of X>` reports the record as not found.
9	—	Execute EXP-SES-001 — UE-initiated attach as a prerequisite procedure.	As defined in EXP-SES-001.

Pass criteria

The restored xControl holds exactly the state of the backup and every account of the backup can log in.
Every Site reconnects within five minutes of xControl starting.
The synchronization check reports exactly the changes made after the backup, and a re-synchronization aligns the Sites to xControl.
A UE attaches and passes traffic after the restore.

Postconditions

Re-create any Mobile Asset or Account that was intentionally lost with the restore.

Evidence collected: xControl screenshot; CLI output; Kubernetes pod and node status.

EXP-OAM-005 — API token expiry warning⚓︎

Attribute	Value
Status	Accepted
Category	Operations
Components	xCore, xControl, xView
Applies to	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
Duration	About 20 min
Prerequisite procedures	EXP-INS-004 — Monitoring path

Objective. An API token that is about to expire raises a warning in xView seven days ahead, and the warning clears once the token is renewed or removed.

Preconditions

Tooling and access
- Access to the xView Prometheus query interface and alert view.
- Log in to xControl with an account that holds the ADMIN role.
Procedures executed first
- EXP-INS-004 — Monitoring path

Procedure

#	Where	Action	Expected result
1	xControl	Create an API token for the Customer with an expiry date less than seven days away (API, Getting Started, generating an API token).	The token is created and its expiry is shown.
2	Tester	Run `xcontrol_token_expiration - time() < 606024*7` in Prometheus and open the alert view.	The token appears in the result and the API-token-expiring alert fires for it at the next evaluation.
3	xControl	Delete the short-lived token (or extend it).	The token is gone or its expiry is more than seven days away.
4	Tester	Watch the alert view.	The alert clears within five minutes.

Pass criteria

A token expiring within seven days raises the warning at the next evaluation, and the warning clears within five minutes of the token being renewed or removed.

Evidence collected: xControl screenshot.

EXP-OAM-006 — Time synchronization and timestamp consistency⚓︎

Attribute	Value
Status	Accepted
Category	Operations
Components	xCore, xControl
Applies to	4G, 5G · Local radio, Public radio · Single node, HA cluster, Geo-redundant
Duration	About 20 min
Prerequisite procedures	EXP-SES-001 — UE-initiated attach

Objective. Every node of the platform is synchronized to a time source, and the timestamps of the same event in the signalling capture, the network function logs and xControl agree.

Preconditions

Tooling and access
- Shell access to the cluster nodes, or kubectl debug on them.
- Start a 3GPP signalling capture (pcap) with your preferred tool on the interfaces the procedure exercises.
- Log in to xControl with an account that can view the Mobile Asset under test.
Procedures executed first
- EXP-SES-001 — UE-initiated attach

Procedure

#	Where	Action	Expected result
1	Infrastructure	On every node, read the time synchronization status (`chronyc tracking` or `timedatectl`).	Each node is synchronized to a source with an offset below 100 milliseconds.
2	—	Execute EXP-SES-001 — UE-initiated attach as a prerequisite procedure.	As defined in EXP-SES-001.
3	Capture	Note the timestamp of the Attach Complete or Registration Complete in the capture.	The timestamp is recorded.
4	Kubernetes	Find the same event in the MME or AMF log.	The log timestamp is recorded.
5	xControl	Open the session record and note its start time.	The start time is recorded.
6	Tester	Compare the three timestamps.	They agree within one second.

Pass criteria

Every node is synchronized with an offset below 100 milliseconds.
The capture, log and xControl timestamps of the same event agree within one second.

Evidence collected: CLI output; 3GPP signalling capture (pcap); xControl screenshot.