Resilience and High Availability⚓︎

Node, pod, database, radio, link and site failures on Expeto Edge (xCore), Expeto Gateway (xRouter) and xControl, including loss of the management path. Tests cover Kubernetes node failover, single-site restart recovery, geo-redundant failover and the isolation of faults between user planes.

EXP-HA-001 — xCore user-plane node failure⚓︎

Objective. Expeto Edge (xCore) on a Kubernetes HA cluster continues to serve Mobile Assets when the node hosting the user-plane services (UPF / PGW-U) fails: the pods restart on the remaining nodes and the device regains service.

Preconditions

• Platform
  • An xCore is deployed on a Kubernetes HA cluster and operational.
  • You have identified the node (VM) hosting the user-plane pods (UPF / PGW-U).
  • Enough nodes remain available to host the user plane after the failure.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
• Tooling and access
  • A tool that measures upload and download throughput from the UE against a server on the enterprise network, such as iperf3 or OpenSpeedTest.

Procedure

Pass criteria

• Every pod of the failed node is running on another node without manual intervention.
• The UE regains service and passes traffic after the failover.

Postconditions

• Record the measured service interruption in the results sheet.

Evidence collected: Kubernetes pod and node status; Speed test result; UE connectivity check.

EXP-HA-002 — xCore session-management node failure⚓︎

Objective. Expeto Edge (xCore) on a Kubernetes HA cluster continues to serve Mobile Assets when the node hosting the session-management services (SMF / PGW-C) fails.

Preconditions

• Platform
  • An xCore is deployed on a Kubernetes HA cluster and operational.
  • You have identified the node (VM) hosting the session-management pods (SMF / PGW-C).
  • Enough nodes remain available to host session management after the failure.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
• Tooling and access
  • A tool that measures upload and download throughput from the UE against a server on the enterprise network, such as iperf3 or OpenSpeedTest.

Procedure

Pass criteria

• Every pod of the failed node is running on another node without manual intervention.
• The UE regains service and passes traffic after the failover.

Postconditions

• Record the measured service interruption in the results sheet.

Evidence collected: Kubernetes pod and node status; UE connectivity check.

EXP-HA-003 — xCore subscriber-data node failure⚓︎

Objective. Expeto Edge (xCore) on a Kubernetes HA cluster continues to authenticate Mobile Assets when the node hosting the local subscriber database (HSS, or UDM, UDR and AUSF) fails: existing sessions continue and new attaches succeed once the pods have restarted.

Preconditions

• Platform
  • An xCore is deployed on a Kubernetes HA cluster and operational.
  • You have identified the node (VM) hosting the subscriber-database pods.
  • Enough nodes remain available to host them after the failure.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
  • A second Mobile Asset, provisioned and its UE powered off.

Procedure

Pass criteria

• Established sessions are unaffected by the failure.
• The subscriber-database pods are ready on another node within five minutes, new attaches succeed once they are, and the subscriber data is complete.

Evidence collected: Kubernetes pod and node status; UE connectivity check; CLI output.

EXP-HA-004 — xCore restart recovery⚓︎

Objective. A non-redundant Expeto Edge (xCore) recovers after every resource hosting it is power-cycled (turned off and back on), and devices regain service without manual intervention.

Preconditions

• Platform
  • An xCore is deployed and operational, and its Site shows as online in xControl.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
• Tooling and access
  • A tool that measures upload and download throughput from the UE against a server on the enterprise network, such as iperf3 or OpenSpeedTest.

Procedure

Pass criteria

• Every service is running after the power cycle without manual intervention.
• The UE reconnects on its own and passes traffic.

Postconditions

• Record the time from power-on to the UE regaining service in the results sheet.

Evidence collected: Service status after recovery; UE connectivity check.

EXP-HA-005 — xCore geographic redundancy⚓︎

Objective. When one xCore of a geo-redundant pair fails, the Mobile Assets it served reconnect to the redundant xCore and continue to receive service.

Preconditions

• Platform
  • Geo-redundant xCores are deployed in an active-active configuration.
• Mobile Asset
  • Mobile Assets are connected to both xCores and passing traffic.
• Tooling and access
  • A tool that measures upload and download throughput from the UE against a server on the enterprise network, such as iperf3 or OpenSpeedTest.

Procedure

Pass criteria

• Every UE of the failed site regains service on the redundant site without manual intervention.

Postconditions

• Record the reconnect time in the results sheet; start the failed site again.

Evidence collected: Speed test result; Measured reconnect time; xControl session event log.

EXP-HA-006 — xRouter geographic redundancy⚓︎

Objective. When one xRouter of a geo-redundant pair fails, the Mobile Assets connected through it reconnect through the redundant xRouter and continue to receive service.

Preconditions

• Platform
  • Geo-redundant xRouters are deployed and operational.
  • Both xRouters are integrated with an xCore.
• Mobile Asset
  • Mobile Assets are connected through both xRouters to the xCore and passing traffic.
• Tooling and access
  • A tool that measures upload and download throughput from the UE against a server on the enterprise network, such as iperf3 or OpenSpeedTest.

Procedure

Pass criteria

• Every UE of the failed xRouter regains service through the redundant one without manual intervention.

Postconditions

• Record the reconnect time in the results sheet; start the failed site again.

Evidence collected: Speed test result; Measured reconnect time.

EXP-HA-007 — xRouter subscriber-management node failure⚓︎

Objective. Expeto Gateway (xRouter) on a Kubernetes HA cluster continues to operate when the node hosting its subscriber-management services fails.

Preconditions

• Platform
  • An xRouter is deployed on a Kubernetes HA cluster and operational.
  • You have identified the node (VM) hosting the subscriber-management pods (AUSF, UDR and UDM for 5G; HSS for 4G).
  • Enough nodes remain available to host subscriber management after the failure.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).

Procedure

Pass criteria

• Every pod of the failed node is running on another node without manual intervention.
• The UE authenticates and connects after the failover.

Evidence collected: Kubernetes pod and node status; UE connectivity check.

EXP-HA-008 — xRouter restart recovery⚓︎

Objective. A non-redundant Expeto Gateway (xRouter) recovers after every resource hosting it is power-cycled (turned off and back on), and devices regain service through it.

Preconditions

• Platform
  • An xRouter is deployed and operational.
  • The xRouter is integrated with an xCore.
• Mobile Asset
  • A Mobile Asset is connected to the xCore through the xRouter and passing data.
• Tooling and access
  • A tool that measures upload and download throughput from the UE against a server on the enterprise network, such as iperf3 or OpenSpeedTest.

Procedure

Pass criteria

• Every service is running after the power cycle without manual intervention.
• The UE reconnects through the xRouter and passes traffic.

Postconditions

• Record the time from power-on to the UE regaining service in the results sheet.

Evidence collected: Service status after recovery; UE connectivity check.

EXP-HA-009 — xControl restart recovery⚓︎

Objective. A non-redundant xControl recovers after every resource hosting it is power-cycled, the xCore sites it manages reconnect to it, and devices on those sites are unaffected throughout.

Preconditions

• Platform
  • An xControl is deployed and operational.
  • An xCore site is deployed and managed by that xControl.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
• Tooling and access
  • You have an xControl account.

Procedure

Pass criteria

• xControl returns to service after the power cycle without manual intervention beyond starting the nodes.
• Devices on the managed sites keep service throughout.
• The Site reconnects and shows online after the restart.

Evidence collected: xControl availability check; Agent reconnect and reconciliation; UE connectivity check.

EXP-HA-010 — xControl unreachable from the xCore⚓︎

Objective. While the xCore cannot reach xControl, existing sessions continue, new devices attach and are authenticated locally, the loss of contact is alarmed, and when the connection returns the Site reconciles without manual action.

Preconditions

• Platform
  • You can block outbound traffic from the xCore to xControl and remove the block again.
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
  • A second Mobile Asset, provisioned and its UE powered off.
• Tooling and access
  • Log in to xControl with an account that holds the ADMIN role.
  • Access to the xView alert view.
• Procedures executed first
  • EXP-INS-002 — Management path and subscriber synchronization

Procedure

Pass criteria

• No impact on existing sessions or on new attaches while xControl is unreachable.
• The loss of contact is alarmed within six minutes.
• The Site is back online within 60 seconds of the connection returning, and the alert clears within five minutes.

Evidence collected: UE connectivity check; xControl screenshot; Agent reconnect and reconciliation.

EXP-HA-011 — Single network function restart⚓︎

Objective. Each network function of the xCore, restarted on its own, recovers within a bounded time, causes no other function to restart, and leaves the core with exactly the sessions it had, with a recorded impact on a device passing traffic.

Preconditions

• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
• Tooling and access
  • Access to the cluster with kubectl.
  • Access to the CLI.
• Procedures executed first
  • EXP-SES-001 — UE-initiated attach

Procedure

Pass criteria

• Every replacement pod is ready within five minutes; no other pod restarts.
• The device regains service within three minutes of each restart without action on the platform.
• The context and session counts return to the baseline after each restart.

Postconditions

• Record the impact per network function (ping loss, re-attach needed or not) in the results sheet.

Evidence collected: Kubernetes pod and node status; UE connectivity check; CLI output.
References: 3GPP TS 29.244 §6.2 (PFCP association and heartbeat procedures); 3GPP TS 23.007 §19 (Restoration of PFCP entities).

EXP-HA-012 — Radio link loss and recovery⚓︎

Objective. When the radio loses its connection to the xCore and later reconnects, the loss is alarmed, the radio re-establishes its association, devices regain service, and no stale context remains.

Preconditions

• Tooling and access
  • Log in to xControl with an account that holds the ADMIN role.
  • Start a 3GPP signalling capture (pcap) with your preferred tool on the interfaces the procedure exercises.
  • Access to the CLI and the xView alert view.
• Procedures executed first
  • EXP-INS-003 — Radio access node connection
  • EXP-SES-007 — Multiple UE attach

Procedure

Pass criteria

• The loss is alarmed and the radio reconnects without manual intervention on the platform.
• Every UE regains service after the radio returns.
• No stale context remains for any UE.

Evidence collected: 3GPP signalling capture (pcap); xControl screenshot; CLI output.
References: 3GPP TS 36.413 §8.7.1 (Reset); 3GPP TS 36.413 §8.7.3 (S1 Setup); 3GPP TS 38.413 §8.7.1 (NG Reset).

EXP-HA-013 — Secure link interruption between xRouter and xCore⚓︎

Objective. A short interruption of the secure link between Expeto Gateway (xRouter) and the public xCore stops user traffic without affecting the device's registration, and traffic resumes when the link returns; a long interruption is detected and sessions are cleaned up.

Preconditions

• Platform
  • You can block traffic on the secure link between the xRouter and the xCore and remove the block again.
• Tooling and access
  • Log in to xControl with an account that holds the ADMIN role.
  • A capture on the secure link (capture point 5).
• Procedures executed first
  • EXP-POL-002 — Enterprise access through the public radio network

Procedure

Pass criteria

• After a 30-second interruption, traffic resumes without a re-attach.
• After a fifteen-minute interruption, service returns within three minutes of the link returning; whether a re-attach was needed is recorded.

Evidence collected: UE connectivity check; 3GPP signalling capture (pcap); xControl session event log.
References: 3GPP TS 23.007 §20 (Path failure handling); 3GPP TS 29.274 §7.1 (Path Management Messages (Echo Request and Response)).

EXP-HA-014 — Database node failure⚓︎

Objective. The loss of one node of the xCore's replicated database does not interrupt service or provisioning, no data is lost, and the database returns to full replication when the node is back.

Preconditions

• Platform
  • The xCore runs its database as a replicated cluster (three members).
• Mobile Asset
  • One Mobile Asset is connected and passing data (execute the UE-initiated attach procedure first if it is not).
• Tooling and access
  • Log in to xControl with an account that holds the ADMIN role.
  • Access to the cluster with kubectl.
• Procedures executed first
  • EXP-SES-001 — UE-initiated attach

Procedure

Pass criteria

• No service or provisioning interruption with one member down.
• No data created during the outage is lost.
• The cluster is back to three synchronized members within ten minutes of the node returning.

Evidence collected: Kubernetes pod and node status; UE connectivity check; CLI output.

EXP-HA-015 — User-plane isolation between Systems⚓︎

Objective. On a site whose Systems have separate user planes, the failure of one System's user plane leaves the other System's sessions untouched, and the failed user plane accepts new sessions once it is back.

Preconditions

• Platform
  • The Site has two Systems, A and B, each served by its own user-plane instance.
• Mobile Asset
  • One Mobile Asset on each System, both connected and passing traffic (EXP-POL-003).
• Tooling and access
  • Access to the cluster with kubectl.
  • Access to the CLI.
• Procedures executed first
  • EXP-POL-003 — System egress mapping

Procedure

Pass criteria

• Zero ping loss on System B while System A's user plane is down.
• System A's user plane is ready within five minutes and accepts an attach within three minutes of being ready.
• No residual session remains.

Evidence collected: Kubernetes pod and node status; UE connectivity check; CLI output.

EXP-HA-018 — Diameter peer failure and failover toward the operator's network⚓︎

Objective. The HSS in Expeto Gateway (xRouter) keeps serving S6a when one of its Diameter connections to the operator's network (DRA links or IPX peers) fails: the Device-Watchdog exchange detects the failure, requests continue over the remaining connection, the failed connection is re-established on its own when transport returns, and the outage raises an alert.

Preconditions

• Platform
  • The HSS in xRouter has at least two Diameter connections toward the operator's network (two entries under diameter.peers, or one DRA reached over two transport paths) and a routing rule under diameter.routingRules that lets requests for the operator's realm use either.
  • You can block the transport of one connection at a time (firewall rule or interface) and remove the block again.
  • Alerting is in place; the DiameterConnectionDown alert exists for the site.
• Mobile Asset
  • The Mobile Asset is Active and attaches through the public network (EXP-POL-002).
• Tooling and access
  • Log in to xControl with an account that holds the ADMIN role.
  • A signalling capture at xRouter (capture point 4) with the display filter diameter.
  • Access to the alert view and to the metric xcore_hss_link_total.
• Procedures executed first
  • EXP-POL-002 — Enterprise access through the public radio network

Procedure

Expected signalling

Capture points: S6a at xRouter (point 4).

Pass criteria

• Watchdog exchanges are observed on every connection, and an unanswered watchdog takes its connection out of service without affecting the other.
• Every attach attempted during the outage of either connection succeeds over the remaining one; no S6a request is lost or left unanswered.
• The failed connection is re-established without manual intervention when its transport returns, and DiameterConnectionDown fires during the outage and clears afterwards.

Postconditions

• Both connections are up and the alert has cleared.

Deployment notes

• Transport multi-homing: Where a connection uses SCTP with a primary and a secondary address (diameter.primaryIp, diameter.secondaryIp), blocking one address exercises SCTP path failover instead: the association stays up, no Diameter-level failover occurs, and requests continue without interruption.
• Single DRA: Where the mobile operator provides one DRA over one transport path, the test is limited to the watchdog detection, the alert and the re-establishment; attaches during the outage fail and are recorded as such.

Evidence collected: 3GPP signalling capture (pcap); UE connectivity check; Service status after recovery.
References: IETF RFC 6733 §5.5 (Transport failure detection (Device-Watchdog)); IETF RFC 6733 §5.6 (Peer state machine); IETF RFC 6733 §6.1 (Diameter request routing overview); 3GPP TS 29.272 §7.1.1 (Use of Diameter base protocol).