Expeto xCore Alerts⚓︎

This page is the reference for the default Prometheus alerts that monitor the Expeto xCore — site connectivity, provisioning synchronization, telecom signalling, and the site database. Each entry shows the alert's severity, how long its condition must hold before it fires, the exact PromQL trigger expression, and what to check when it fires.

Before using this page, read How to read these alerts for severity levels and the conventions shared by all alert expressions, and How alerts work for how alerts are defined and how tiers escalate. The metrics used in these expressions are described in xCore Metrics.

xCore connectivity⚓︎

These alerts cover the link between each site's Expeto Agent and xControl. They are the first alerts to look at when a site appears to be offline.

xCoreAgentOutOfContact⚓︎

Escalation: if the condition persists, xCoreAgentOutOfContactTier2 after 1h and xCoreAgentOutOfContactTier3 after 4h — see alert tiers.

Trigger condition

up{job="site",site_state!~"decommissioned|maintenance|poc",tier=~"1|"} == 0 or up{job="xAgent",site_state!~"decommissioned|maintenance|poc",tier=~"1|"} == 0

The Expeto Agent at the affected site has stopped reporting metrics. The agent may be down, or it may have lost its network connection to xControl. This is the primary "site out of contact" alert.

What to check: Confirm whether the site is reachable at all (VPN/IPsec tunnel, WAN link), then whether the agent process and its containers are running on the site host. If the agent is up but unreachable, investigate the network path between the site and xControl.

ExpetoNetworkFunctionDown⚓︎

Trigger condition

xcore_nf_node_up{status="unhealthy", site_state!~"decommissioned|maintenance"} == 0

The site's agent cannot communicate with one of the network functions it manages (the function and its address are reported in the alert's labels). The function may be down or in a bad state.

What to check: Look at the health and logs of the named network function, and verify it is reachable from the agent host. Restore or restart the function if it has failed.

xCoreDBNotSyncing⚓︎

Trigger condition

xcore_db_sync_delta_seconds{site_state!~"decommissioned|maintenance"} > 75

The site's local database has not synchronized with xControl for more than 75 seconds. Provisioning changes made in xControl are not reaching the site.

What to check: Verify the agent's connection to xControl and look for a message backlog (see JMSQueueNotDraining). If connectivity is healthy, check the agent logs for database errors at the reported path.

AgentJmsMessagesDelayed⚓︎

Trigger condition

avg by (customer, site) (xcore_jms_delay_time{site_state!~"decommissioned|maintenance", quantile="0.95"} / 1000) > 150

The site's agent is not processing messages from xControl fast enough: the 95th-percentile message delay exceeds 150 seconds. Provisioning changes are applied late.

What to check: Check the agent host for resource exhaustion (CPU, memory, disk I/O) and the quality of the site's network link. A large provisioning burst can also cause a temporary backlog that clears on its own.

xCore synchronization⚓︎

These alerts compare the intended state held by xControl with what is actually provisioned in the site's network functions. A mismatch means an object exists but its data differs, missing means an expected object is absent, and extra means an object exists that xControl does not expect. All three point to the same follow-up: inspect the affected site's synchronization status in xControl and re-synchronize.

HssSubscriberMismatch⚓︎

Trigger condition

xcore_sync_mismatch{site_state!~"decommissioned|maintenance",component="hss"} > 0

The site's HSS holds subscribers whose provisioned data does not match what xControl expects. Affected Mobile Assets may behave inconsistently.

What to check: Open the site's synchronization view in xControl to identify the affected subscribers and trigger a re-sync. If the mismatch recurs, look for an external system writing to the HSS directly.

HssSubscriberMissing⚓︎

Trigger condition

xcore_sync_missing{site_state!~"decommissioned|maintenance",component="hss"} > 0

Subscribers that should be provisioned in the site's HSS are absent. The affected Mobile Assets cannot attach to the network at this site.

What to check: Re-synchronize the site from xControl. If subscribers stay missing, check the agent logs for provisioning errors on the reported instance.

HssSubscriberExtra⚓︎

Trigger condition

xcore_sync_extra{site_state!~"decommissioned|maintenance",component="hss"} > 0

The site's HSS holds subscribers that xControl does not expect — typically left over from deletions that did not propagate, or from provisioning outside of xControl.

What to check: Review the extra entries in the site's synchronization view and remove them via a re-sync. Investigate how they were created if the alert recurs.

MmePrivateRadioMissing⚓︎

Trigger condition

xcore_sync_missing{site_state!~"decommissioned|maintenance",component="privateradio"} > 0

One or more private radios that should be configured in the site's MME are absent.

What to check: Re-synchronize the site from xControl and confirm the radios reappear. The alert fires on a single missing radio once the condition has held for two minutes, so it clears on its own when a re-sync completes; if it persists, the radio is not being provisioned.

MmePrivateRadioMismatch⚓︎

Trigger condition

xcore_sync_mismatch{site_state!~"decommissioned|maintenance",component="privateradio"} > 0

The site's MME has private radios whose configuration does not match what xControl expects.

What to check: Identify the affected radios in the site's synchronization view and re-sync. Check for local configuration changes made outside of xControl.

MmePrivateRadiosExtra⚓︎

Trigger condition

xcore_sync_extra{site_state!~"decommissioned|maintenance",component="privateradio"} > 0

The site's MME has private radios configured that xControl does not expect.

What to check: Review and remove the unexpected radios via a re-sync from xControl.

HssKeysetMissing⚓︎

Trigger condition

xcore_sync_missing{site_state!~"decommissioned|maintenance",component="site"} > 0

Authentication keysets that should be provisioned at the site are absent. Mobile Assets relying on those keysets cannot authenticate, which is why this is critical rather than a warning.

What to check: Re-synchronize the site from xControl immediately and confirm the keysets are restored. Check agent logs for provisioning errors if they stay missing.

HssKeysetMismatch⚓︎

Trigger condition

xcore_sync_mismatch{site_state!~"decommissioned|maintenance",component="site"} > 0

Keysets at the site do not match the values xControl expects. Authentication for the affected Mobile Assets can fail.

What to check: Re-synchronize the site from xControl and verify the mismatch count returns to zero.

HssKeysetExtra⚓︎

Trigger condition

xcore_sync_extra{site_state!~"decommissioned|maintenance",component="site"} > 0

The site holds keysets that xControl does not expect.

What to check: Review the unexpected keysets and remove them via a re-sync; investigate their origin, since unexpected credentials are a security concern.

DiameterConnectionDown⚓︎

Trigger condition

((xcore_hss_link_total{site_state!~"decommissioned|maintenance", type="diameter", status="opened"} - on (customerId, siteId, instance, nfnode, nfnodeip, type) xcore_hss_link_count{site_state!~"decommissioned|maintenance"}) != 0) or ((xcore_hss_link_total{site_state!~"decommissioned|maintenance", type="diameter", status="opened"} unless on (customerId, siteId, instance, nfnode, nfnodeip, type) xcore_hss_link_count{site_state!~"decommissioned|maintenance"}) > 0)

The number of open Diameter links to a peer differs from the expected link count — one or more Diameter connections (4G/5G signalling, e.g. S6a) are down.

What to check: Check the reachability and status of the Diameter peer named in the alert labels, and the SCTP/TCP transport between the site and that peer. Signalling to and from the affected peer is impaired while the link is down.

M3UAConnectionDown⚓︎

Trigger condition

((xcore_hss_link_total{site_state!~"decommissioned|maintenance", type="map", status="opened"} - on (customerId, siteId, instance, nfnode, nfnodeip, type) xcore_hss_link_count{site_state!~"decommissioned|maintenance"}) != 0) or ((xcore_hss_link_total{site_state!~"decommissioned|maintenance", type="map", status="opened"} unless on (customerId, siteId, instance, nfnode, nfnodeip, type) xcore_hss_link_count{site_state!~"decommissioned|maintenance"}) > 0)

The number of open M3UA links (3G MAP signalling over SS7) differs from the expected count — a 3G signalling connection is down. The long 31-minute hold time tolerates routine link flaps before alerting.

What to check: Check the M3UA/SCTP association to the peer named in the alert labels and the health of the SS7 signalling path. 3G authentication and mobility for the affected peer are impaired while the link is down.

SmppConnectionDown⚓︎

Trigger condition

((xcore_hss_link_total{site_state!~"decommissioned|maintenance", type="smpp", status="opened"} - on (customerId, siteId, instance, nfnode, nfnodeip, type) xcore_hss_link_count{site_state!~"decommissioned|maintenance"}) != 0) or ((xcore_hss_link_total{site_state!~"decommissioned|maintenance", type="smpp", status="opened"} unless on (customerId, siteId, instance, nfnode, nfnodeip, type) xcore_hss_link_count{site_state!~"decommissioned|maintenance"}) > 0)

The number of open SMPP links differs from the expected count — an SMS signalling connection is down. SMS delivery through the affected peer is interrupted.

What to check: Check the SMPP peer named in the alert labels and the transport between the site and that peer.

PrivateRadioConnectionDown⚓︎

Trigger condition

abs( max by (customer, customerId, site, siteId, site_state) (xcore_sync_correct{component="privateradio",site_state!~"decommissioned|maintenance|poc"}) + max by (customer, customerId, site, siteId, site_state) (xcore_sync_missing{component="privateradio",site_state!~"decommissioned|maintenance|poc"}) - max by (customer, customerId, site, siteId, site_state) (xcore_privateradio_count{site_state!~"decommissioned|maintenance|poc"}) ) > 0

The number of private radios actually connected at the site differs from the number that should be connected.

What to check: Compare the site's expected radio inventory in xControl against what is connected, then check the disconnected radios' power and transport links.

xCore telecom signalling⚓︎

These are traffic-pattern alerts: they fire when a site that normally carries signalling traffic (more than 1000 requests over the past day) suddenly goes quiet. They are informational — silence may be a real upstream problem or an expected lull.

Low3gMapAuthRequests⚓︎

Trigger condition

(increase(xcore_hss_map_messages_total{message="send_authentication_info",type="request",site_state!~"decommissioned|maintenance"}[1d]) > 1000 and increase(xcore_hss_map_messages_total{message="send_authentication_info",type="request",site_state!~"decommissioned|maintenance"}[15m]) == 0)

A site that normally handles 3G MAP authentication traffic has received none in the last 15 minutes.

What to check: Confirm whether the silence is expected (maintenance, traffic moved elsewhere). If not, check the 3G signalling path — see M3UAConnectionDown — and the upstream network feeding the site.

Low4gS6aAuthRequests⚓︎

Trigger condition

(increase(xcore_hss_s6_messages_total{message=~"authentication_information",type=~"request",site_state!~"decommissioned|maintenance"}[1d]) > 1000 and increase(xcore_hss_s6_messages_total{message=~"authentication_information",type=~"request",site_state!~"decommissioned|maintenance"}[15m]) == 0)

A site that normally handles 4G S6a authentication traffic has received none in the last 15 minutes.

What to check: Confirm whether the silence is expected. If not, check the Diameter signalling path — see DiameterConnectionDown — and the upstream 4G core.

Database health⚓︎

These alerts monitor the MariaDB databases used by xCore sites, via the metrics of the mariadb scrape job (see Monitored Endpoints).

MariadbDown⚓︎

Trigger condition

mysql_up{site_state!~"decommissioned|maintenance"} != 1

A MariaDB instance is down or unreachable by its metrics exporter.

What to check: Check whether the database process/container is running on the reported instance and review its logs for the cause of the failure (crash, out of disk, restart loop).

MariadbTooManyConnections⚓︎

Trigger condition

max_over_time(mysql_global_status_threads_connected{site_state!~"decommissioned|maintenance"}[1m]) / mysql_global_variables_max_connections{site_state!~"decommissioned|maintenance"} * 100 > 80

The instance is using more than 80% of its configured max_connections. If the limit is reached, new connections are refused and dependent services fail.

What to check: Identify which clients hold the connections (SHOW PROCESSLIST) and look for connection leaks. Raise max_connections only if the demand is legitimate.

MariadbTooManyOpenFiles⚓︎

Trigger condition

(mysql_global_status_open_files{site_state!~"decommissioned|maintenance"} / mysql_global_variables_open_files_limit{site_state!~"decommissioned|maintenance"} * 100) > 2

The instance's open file count is above the expected share of its open_files_limit. The 2% threshold is deliberately low to act as an early warning of unusual file usage.

What to check: Compare the open file count against the instance's normal baseline. If it is climbing, look for table growth, missing cleanup, or a lowered open_files_limit.

MariadbTooManyOpenInnodbFiles⚓︎

Trigger condition

(mysql_global_status_innodb_num_open_files{site_state!~"decommissioned|maintenance"} / mysql_global_variables_open_files_limit{site_state!~"decommissioned|maintenance"} * 100) > 2

Same early-warning check as MariadbTooManyOpenFiles, but for files opened by the InnoDB storage engine specifically.

What to check: As for MariadbTooManyOpenFiles; additionally check for a large number of InnoDB tablespaces.

MariadbHighPreparedStatementsUtilization⚓︎

Trigger condition

max_over_time(mysql_global_status_prepared_stmt_count{site_state!~"decommissioned|maintenance"}[1m]) / mysql_global_variables_max_prepared_stmt_count{site_state!~"decommissioned|maintenance"} * 100 > 80

The instance is using more than 80% of max_prepared_stmt_count. Hitting the limit causes application errors on statement preparation; a steadily climbing count usually indicates a statement leak in a client.

What to check: Identify the client application holding prepared statements and restart or fix it. Raise the limit only for legitimate sustained demand.

MariadbHighThreadsRunning⚓︎

Trigger condition

max_over_time(mysql_global_status_threads_running{site_state!~"decommissioned|maintenance"}[1m]) / mysql_global_variables_max_connections{site_state!~"decommissioned|maintenance"} * 100 > 60

An unusually high share of connections are actively executing at once. This usually means queries are running slowly and piling up rather than completing.

What to check: Look at the currently running queries (SHOW PROCESSLIST) for long-running or blocked statements, and at host CPU and disk I/O. Often accompanies MariadbSlowQueries.

MariadbSlowQueries⚓︎

Trigger condition

increase(mysql_global_status_slow_queries{site_state!~"decommissioned|maintenance"}[1m]) > 40

The instance is logging more than 40 slow queries per minute.

What to check: Review the slow query log to identify the offending statements, then check for missing indexes, lock contention, or resource pressure on the host.

MariadbInnodbLogWaits⚓︎

Trigger condition

rate(mysql_global_status_innodb_log_waits{site_state!~"decommissioned|maintenance"}[15m]) > 10

Writes are waiting for InnoDB redo-log space at a sustained rate — the log is being written faster than it can be flushed.

What to check: Check disk write latency on the host (see DiskIOUnhealthy); if the disk is healthy, the InnoDB log buffer/file size may be too small for the write load.

MariadbClusterOutOfSync⚓︎

Trigger condition

abs(count by(customer,site) (sum by (customer,site,statefulset) (kube_statefulset_status_replicas{site_state!~"decommissioned|maintenance",job="k8s",statefulset=~".+-mariadb-galera"})) - on(customer, site) count by (customer,site) (count by(customer, site, wsrep_cluster_state_uuid) (mysql_galera_status_info{site_state!~"decommissioned|maintenance",job="mariadb-galera"}))) > 0

The MariaDB Galera cluster at a site has partitioned: its members no longer agree on a single cluster state. Writes may diverge between partitions.

What to check: Inspect the Galera status (wsrep_cluster_status, wsrep_cluster_size) on each member and the network between them. Follow the standard Galera recovery procedure to rejoin or bootstrap members — do not restart members blindly.

MariadbInstanceOutOfSync⚓︎

Trigger condition

label_replace(mysql_galera_status_info{site_state!~"decommissioned|maintenance"}, "__uuid__", "$1", "wsrep_cluster_state_uuid", "(.*)") != on (customer,site,instance,__uuid__) label_replace(mysql_galera_status_info{site_state!~"decommissioned|maintenance"}, "__uuid__", "$1", "wsrep_local_state_uuid", "(.*)")

A single Galera member's local state no longer matches the cluster state — the instance has fallen out of sync while the rest of the cluster is healthy.

What to check: Check the desynchronized member's Galera logs and let it complete a state transfer (IST/SST) to rejoin. If it cannot rejoin, remove and re-add it to the cluster.

Experimental alerts⚓︎

These alerts are under evaluation. They are enabled by default, but their thresholds and severities are still being tuned against real deployments, and any of them may change or be withdrawn in a future release. Treat them as signals worth investigating rather than as settled operational policy — and tell Expeto support if one proves noisy in your environment.

xCoreCrashRestart⚓︎

Trigger condition

resets(xcore_uptime_minutes{site_state!~"decommissioned|maintenance"}[5m]) > 0

The site's uptime counter went backwards, which means the xCore restarted. The alert cannot tell a planned restart from a crash, which is why it is informational: during a maintenance window it is expected, and outside one it is not.

What to check: Establish whether the restart was planned. If it was not, look at the logs from before the restart for the cause, and check whether the process was killed for exceeding a memory limit. A site that restarts repeatedly will show this alert clearing and re-firing.

ProbeFailure⚓︎

Trigger condition

probe_success == 0

A synthetic probe failed: the blackbox exporter could not reach the target it was asked to check. Unlike a scrape failure, this alert reports on a service the platform depends on rather than on a metrics endpoint, so it fires even when every Expeto component is healthy.

What to check: The failing target is in the alert's instance label. Verify the target is reachable from the site — by name as well as by address, since probe failures are often DNS — and confirm any firewall or proxy on the path still permits the check.

ProbeSuccessLow⚓︎

Trigger condition

avg_over_time((probe_success >bool 0)[5m:1m]) * 100 < 85

A synthetic probe is succeeding less than 85 percent of the time over five minutes. The target is reachable but unreliable, which ProbeFailure does not catch because the probe keeps recovering between checks.

What to check: The same path as for ProbeFailure, but look for intermittent causes rather than an outage: packet loss on the link, an overloaded target, or one unhealthy member behind a load balancer.