Platform End-to-End Suites⚓︎
The platform end-to-end suites install the Expeto Platform from its published Helm charts onto a real Kubernetes cluster, provision a tenant through the xControl API, drive subscriber sessions through 3GPP peer simulators, inject faults, measure capacity and stability, and verify that the cluster is left clean. They run on every merge request to the platform verification repository, and a weekly deep soak runs every session flow at full size for long enough to judge memory behaviour.
How a suite runs⚓︎
- Preflight. The cluster identity is checked against an allow-list and the cluster is confirmed clean of earlier runs.
- Install. xControl is installed first; the suite's cores follow, each into its own namespace labelled with the run's identity. A tenant (Customer, Site, tokens) is provisioned through the xControl API and the resulting parameters are injected into the core installs, exactly as an MNO would wire a new site.
- Wait. Every workload, database cluster and, for the Kubernetes-operator managed core, every Site resource must report ready before the first test runs.
- Test. The suite's test classes run in a fixed order in one JVM. Session flows are single runs with hard assertions; soaks and capacity ramps count every outcome and fail on any that is not a success.
- Tear down and verify. Charts are uninstalled, namespaces removed, and the cluster is checked clean again. On any failure the state of every managed namespace is captured first.
The framework is written in Kotlin on the Citrus integration-test framework and JUnit 5. Radio and peer-core signalling comes from Expeto's 3GPP peer simulator: for the 4G split core it plays two eNodeBs over SCTP and S1AP with a fleet of UEs; for the Kubernetes-operator managed core it plays the S11 peer. Echo services on the N6 side answer the UEs' traffic so that byte counts are exact. Browser checks of xControl use Playwright driving Chromium against the deployed portal.
The suites⚓︎
| Suite | Installs on top of xControl | Radio or load generator | Test classes | Cadence | Wall clock |
|---|---|---|---|---|---|
XControlSuite |
Nothing more; a placeholder Site gives site tests a target | — | 7 | Every merge request | 10 to 25 min |
XControlAndXViewSuite |
xView | — | 8 | Every merge request | 10 to 25 min |
FullSuite |
xView and the 5G core at chart defaults (AMF, UDM, UDR, AUSF, one SMF, one UPF) | — | 11 | Every merge request | Under 1 h |
TaigaSuite |
The Kubernetes-operator managed core: one 4G user plane, a UE pool of 65 000 addresses, 12 000 sessions, exact usage-record checks | Two simulator instances as S11 peers, one N6 echo service | 12 | Every merge request; weekly deep soak | About 50 min |
TaigaSplitSuite |
The Kubernetes-operator managed core with a split user plane: an SGW-U and two PGW-U instances, one serving a dynamically addressed System and one a statically addressed System | Same fleet, one echo service per user plane | 15 | Every merge request; weekly deep soak | About 65 min |
PrivateLteSuite |
The 4G split core: MME and HSS, SGW-C and PGW-C as two session-management instances, SGW-U and PGW-U as two user planes, 11 000 sessions | One simulator instance playing two eNodeBs over S1AP | 14 | Every merge request; weekly deep soak | About 2 h |
XCoreSuite, PrivateLteHandoverSuite |
The 5G core alone; the 4G split core without its three long-running tests | — / two eNodeBs | 8 / 12 | Engineering workstations | — |
The per-merge-request leg runs every suite on the default Kubernetes version. Legs for an older and a newer Kubernetes version are present in every pipeline as manual, non-blocking jobs and are run before a release.
Shared checks⚓︎
Run in every suite.
| Check | Asserts |
|---|---|
| Container image validation | Every container and init container in every managed namespace runs an image from the Expeto registry, tagged with the chart version, with imagePullPolicy: IfNotPresent; third-party images are on an allow-list. |
| Helm template invariance | Rendering the installed chart twice with the same arguments and the live cluster capabilities gives the same output. |
| Provisioning | The persisted run state exists, the xControl release is deployed with the expected chart, the administrator credentials exist, and every Site resolves by identifier to its provisioned name through the ingress. |
| Vulnerability scan | Trivy scans every managed namespace for HIGH and CRITICAL findings and produces JSON, table, HTML and JUnit reports on every run; the findings are reviewed with the software bill of materials before a release. |
| Documentation sites | A browser opens the public and administrator documentation served by the deployment: HTTP 200, the right sections present and absent per audience, a sample of links resolving. |
| Single sign-on | A user that exists only in the identity provider logs in; an xControl account is created on first login and the redirect chain passes through the identity provider and ends in the authenticated application. |
| Device data | No device lookup is ever purchased from an external service during a run: the device data shipped in the xControl image resolves every handset the simulators present. |
Stack checks⚓︎
| Check | Suite | Asserts |
|---|---|---|
| xView dashboards | xView suites | Grafana holds exactly the expected production dashboards; every dashboard renders and every dashboard link is followed and must resolve. |
| Site status in xControl | 5G core | A browser opens the Site's detail page: the status is not Offline and the agent version matches the chart version. |
| Mobile Asset provisioning | 5G core | A Mobile Asset provisioned through the API (APN, System Profile, APN Profile, System, Mobile Asset) is listed by the UDR and by the SMF within 120 seconds, and the agent's full synchronization check reports no discrepancy for it. |
| Metrics federation | 5G core | The core's Prometheus federation endpoint serves at least 40 000 samples and 200 xcore_* series, every one of which is fresh in xView within 10 minutes and carries the customer, customerId, site and siteId labels of the provisioned tenant. |
| Site resources | Kubernetes-operator managed core | Every Site resource is Ready with an observed generation at least its generation, the agent is ready, and the Site resolves in xControl. In the split configuration, exactly three user-plane resources (the SGW-U and two PGW-U instances) and exactly two Systems exist. |
Session flows⚓︎
A flow is a named sequence of operations that a single-run test plays once with hard assertions and that the soak engine replays at volume. Every flow first provisions its Mobile Asset through xControl and waits for Provisioned; on the 4G split core it also takes a settled census of MME contexts, session-management sessions and user-plane sessions, and ends by asserting that every gauge is exactly back to that baseline within 2 minutes.
| Core | Flow | Sequence | Asserts |
|---|---|---|---|
| 4G split core | attach-detach |
attach, five pings of exact size, detach | The simulator reports an established session with a UE address and the MME attach counter moves within 2 minutes; xControl usage is exactly 185 bytes down and 185 bytes up (five pings of 37 bytes), fed from usage records within 3 minutes; the detach is accepted, active sessions decrease and the detach counter moves within 30 seconds. |
| 4G split core | idle-reactivate |
attach, pings, idle, reactivate, census unchanged, ping, detach | The idle counter moves; a Service Request reactivates the UE and the active counter moves; the census is identical to the post-attach census (no context minted or dropped); the session is still held. |
| 4G split core | idle-detach |
attach, pings, idle, detach | The terminal detach is issued against an idle UE; this is the coverage of the wake-then-detach path. |
| 4G split core | s1-handover |
attach, pings, S1 handover, census unchanged, ping, detach | The handover is accepted, the serving eNodeB reported by the simulator differs from the one at attach, the census is identical, the session survives. |
| 4G split core | x2-handover |
attach, pings, X2 handover (the MME sees only the Path Switch Request), census unchanged, ping, detach | As for the S1 handover. |
| Kubernetes-operator managed core | attach-detach |
attach, UE address from the System's pool, session in the SMF, three pings, Connected, detach, Disconnected, usage, record fields | The UE address is inside the System's pool (or equals the static IP address); the SMF holds the subscriber within 120 seconds; three of three ICMP replies arrive over GTP-U from the System's N6 echo; xControl shows Connected and the SMF session is active, then Disconnected and the SMF row is gone; usage is exactly 111 bytes each way; the session record carries IMSI, MSISDN, bearer, UE address and APN; the history carries the visited MCC, MNC and tracking area. |
| Kubernetes-operator managed core | reattach |
the sequence above, twice | Both sessions come up on the same Mobile Asset; at least two closed session-history records exist; no residual SMF session. |
Each flow runs once per user plane on the split configuration, so every assertion is made for the dynamically addressed and the statically addressed System.
Fault injection and semantics⚓︎
| Test | Does | Asserts |
|---|---|---|
| System reassignment | A Mobile Asset with a live session on the dynamically addressed System is moved to the statically addressed System, with a static IP address, while Connected; then detached and re-attached | An explicit detach clears the SMF session within 60 seconds; the re-attach lands exactly on the new static address with three of three pings on the new user plane, Connected then Disconnected; no residual SMF session. |
| Static IP semantics | A static-IP session is held while a second Mobile Asset attaches with the same pinned address and a third with an address outside the pool | No two active sessions ever hold the same UE address; an out-of-pool address is never assigned; the baseline session still pings after both attempts. |
| User-plane failure isolation | Two static-IP sessions are held on user plane B while user plane A's pod is deleted; a Mobile Asset then re-attaches on A | Every held B session keeps receiving ping replies during A's outage; A's pod is ready again within 5 minutes and accepts an attach within 3 minutes with an in-pool address that pings; no residual sessions and no half-open legs on the SGW-U. |
| Radio capability sweep | Attach with ascending S1AP UE radio capability sizes from 512 to 11 000 bytes on three paths: attach, S1 handover container, post-X2 | Each path tolerates at least 1 024 bytes; the first break per path is reported. |
Soaks and capacity⚓︎
Soaks judge stability; single runs judge correctness. A soak plays flows at volume and then applies, in order, the verdicts below. Baselines are taken after warm-up and before load: the claim is "back to where we started". The report and the memory summary are written before any verdict is asserted, and any verdict that does not hold fails the run.
| # | Verdict | Holds when |
|---|---|---|
| 1 | No crash | No watched container restarted (4G split core: MME, HSS, SGW-C, PGW-C, SGW-U, PGW-U, agent, database, simulator; Kubernetes-operator managed core: every running pod in the core namespace and the simulator fleet). |
| 2 | Did churn | The run produced the number of attaches its duration and concurrency imply; a stalled load generator is a failure, not a pass. |
| 3 | Load generator drained | Every session the simulator opened was released, and its live-session count is back to the pre-run value. |
| 4 | Every session succeeded | Every attach, every idle, reactivation and handover procedure, and every detach completed; the simulator lost no session. |
| 5 | No session leak | Every census gauge (MME contexts, session-management sessions, user-plane sessions; SMF session table and per-System residual by UE pool on the Kubernetes-operator managed core) is exactly back to its baseline. |
| 6 | No memory growth | The memory of every core container is flat after warm-up: no upward trend over the second half of the run. |
| 7 | Core-specific hooks | Kubernetes-operator managed core: the simulator's attach and detach ledger equals the engine's counts; no Mobile Asset is left Connected; usage-record byte totals equal pings times 37 bytes. |
| Run | Load | Verdict |
|---|---|---|
| Mixed-flow soak, 4G split core | 100 live sessions, 10-second lifetime, a random registered flow per session, 200 IMSIs | The verdict set above. |
| Mixed-flow soak, Kubernetes-operator managed core | 300 Mobile Assets, 300 live sessions, lifetimes of 0.5 to 2 seconds, at least 6 000 sessions, 64 calls in flight, keep-alive pings | The verdict set above with the core-specific hooks. |
| Weekly deep soak | Every registered flow in isolation for one hour at full size: private LTE 100 live sessions on 200 subscribers across its 5 flows; Kubernetes-operator managed core 300 live sessions on 300 subscribers across its 2 flows, once per user plane. Every merge request runs the same soaks as a 60-second smoke. | The verdict set above; the memory verdict is decisive at this length. |
| Unknown-IMSI signalling storm, 4G split core | 250 IMSIs on an unserved PLMN hammer the MME for 5 to 10 minutes, concurrency ramping from 2 to 32 while MME CPU stays under 4 cores | No MME, HSS or simulator restart; at least 100 attempts, every one rejected, zero accepted; an unknown IMSI is still refused without timing out afterwards; MME memory returns to its baseline; attached, active and bearer gauges are back to baseline. |
| Maximum concurrent sessions, 4G split core | 10 800 Mobile Assets provisioned, then a ramp to 10 000 held sessions with 32 in flight, held 3 minutes, released | Every Mobile Asset is reconciled into the HSS; no watched container restarted; the simulator drained; every release answered; MME, both session-management and both user-plane instances back to baseline; the exact target reached. |
| Session capacity ramp, Kubernetes-operator managed core | 5 000 Mobile Assets with two sessions each (10 000 sessions, 5 000 held), up to 50 attaches per second, 128 in flight | No watched container restarted while holding; the simulator back to its pre-run live count; every release answered; the target reached, otherwise the run fails naming the peak, the ceiling reason and per-component peak memory and CPU. |
A run on the Kubernetes-operator managed core that detects a wedge (a crashed container or a stalled throughput) freezes instead of cleaning up, leaving Mobile Assets, sessions and echo services in place for diagnosis. A stall dominated by transport errors is reported as a load-generator fault, never as a platform defect.
Acceptance tests mirrored by automation⚓︎
The acceptance test catalog cites, on each test card, the automated checks that exercise the same behaviour. The table below is the reverse view.
| Automated check | Acceptance tests it mirrors |
|---|---|
Platform suite, 4G split core: session flow attach-detach (single run, mixed soak and weekly deep soak) |
EXP-SES-001, EXP-SES-002, EXP-SES-008 |
Platform suite, 4G split core: session flow idle-reactivate |
EXP-SES-004, EXP-SES-005 |
Platform suite, 4G split core: session flow idle-detach |
EXP-SES-004 |
Platform suite, 4G split core: session flow s1-handover |
EXP-MOB-002 |
Platform suite, 4G split core: session flow x2-handover |
EXP-MOB-001 |
| Platform suite, 4G split core: mixed-flow soak (100 live sessions, random flow per session) and weekly deep soak | EXP-CAP-004, EXP-SES-007 |
| Platform suite, 4G split core: unknown-IMSI signalling soak (250 unserved IMSIs) | EXP-PRV-005, EXP-SEC-008 |
| Platform suite, 4G split core: 10 000 concurrently held sessions | EXP-CAP-001 |
Platform suite, Kubernetes-operator managed core: session flow attach-detach, once per user plane |
EXP-SES-001, EXP-SES-002 |
Platform suite, Kubernetes-operator managed core: session flow reattach |
EXP-SES-008 |
| Platform suite, Kubernetes-operator managed core: 10 000-session capacity ramp | EXP-CAP-001 |
| Platform suite, split user plane: System reassignment of a connected Mobile Asset (Cancel Location with re-attach) | EXP-PRV-007 |
| Platform suite, split user plane: static IP semantics (duplicate and out-of-pool addresses) | EXP-PRV-009 |
| Platform suite, split user plane: one user-plane instance deleted while sessions are held on the other | EXP-HA-001, EXP-HA-011, EXP-HA-015 |
| Platform suite, 5G core: Mobile Asset provisioned through the xControl API appears in UDR and SMF within 120 s | EXP-INS-002, EXP-PRV-001 |
| Release certification: subscriber lifecycle scenarios on the reference 5G configuration | EXP-SES-001 |