Expeto xControl Alerts⚓︎
This page is the reference for the default Prometheus alerts that monitor Expeto xControl — its messaging to sites, its external integrations, and its PostgreSQL database. Each entry shows the alert's severity, how long its condition must hold before it fires, the exact PromQL trigger expression, and what to check when it fires.
Before using this page, read How to read these alerts for severity levels and the conventions shared by all alert expressions, and How alerts work for how alerts are defined and how tiers escalate.
| Alert | Severity | Fires after |
|---|---|---|
| JMSQueueNotDraining | Critical | 5m |
| LocationLookupApiCreditsLow | Warning | Immediately |
| ImeiLookupCreditsLow | Warning | Immediately |
| IpsecTunnelDown | Critical | 2m |
| SiteApiTokenExpiringSoon | Warning | Immediately |
| PostgresDeadlocks | Critical | 1m |
| PostgresTxDurationTooLong | Warning | Immediately |
| PostgresClusterOutOfSync | Critical | 2m |
| PostgresCollectorUnhealthy | Critical | 1m |
xControl health⚓︎
JMSQueueNotDraining⚓︎
| Severity | Fires after |
|---|---|
| Critical | 5m |
Escalation: if the condition persists, JMSQueueNotDrainingTier2 after 1h and JMSQueueNotDrainingTier3 after 4h — see alert tiers.
Trigger condition
xcontrol_queue_health{site_state!~"decommissioned|maintenance",tier=~"1|",siteAlive="1"} == 100
The message queue from xControl to a site is not draining: xControl is producing messages for the site but the site's Expeto Agent is not consuming them. The agent may be down or may have lost its network connection to xControl. Note the siteAlive="1" filter — the alert only fires for sites that are otherwise considered alive, which points at a stuck queue rather than a dead site.
What to check: Check the site's agent first — if xCoreAgentOutOfContact is firing for the same site, chase that. If the agent is up, inspect its logs for message-processing errors and check the network path between the site and xControl.
xControl miscellaneous⚓︎
LocationLookupApiCreditsLow⚓︎
| Severity | Fires after |
|---|---|
| Warning | Immediately |
Trigger condition
xcontrol_cpsapi_credits < 5000
xControl has fewer than 5000 credits left with the external location-lookup API provider. When the credits run out, location lookups fail.
What to check: Review recent lookup volume for anything unusual, and top up the credits with the provider before they are exhausted.
ImeiLookupCreditsLow⚓︎
| Severity | Fires after |
|---|---|
| Warning | Immediately |
Trigger condition
xcontrol_imeilookup_credits < 2000
xControl has fewer than 2000 credits left with the external IMEI-lookup API provider. When the credits run out, IMEI lookups fail.
What to check: As for LocationLookupApiCreditsLow: review usage and top up before exhaustion.
IpsecTunnelDown⚓︎
| Severity | Fires after |
|---|---|
| Critical | 2m |
Trigger condition
expeto_ipsec_tunnel_health == 0
An IPsec tunnel endpoint has been unpingable for 2 minutes and the tunnel is probably down. The alert's labels report the tunnel's configuration file and peer IP. All traffic routed through the tunnel — typically site-to-platform traffic — is affected.
What to check: Check the tunnel status on the IPsec concentrator, the reachability of the peer gateway, and whether the underlying network path between the two endpoints is up.
SiteApiTokenExpiringSoon⚓︎
| Severity | Fires after |
|---|---|
| Warning | Immediately |
Trigger condition
(xcontrol_token_expiration{site_state!~"decommissioned|maintenance"} - time()) < (60 * 60 * 24 * 7)
An API token known to xControl expires within the next 7 days. If it expires, every integration that authenticates with it starts failing.
What to check: Identify the token from the alert's labels, renew or rotate it, and update the systems that use it before the expiry date.
Database health⚓︎
These alerts monitor the PostgreSQL database behind xControl, via the metrics of the postgres scrape job (see Monitored Endpoints).
PostgresDeadlocks⚓︎
| Severity | Fires after |
|---|---|
| Critical | 1m |
Trigger condition
increase(pg_stat_database_deadlocks{datname!~"template.*|postgres"}[1m]) > 5
A database is recording more than 5 deadlocks per minute — concurrent transactions are repeatedly blocking each other, and PostgreSQL is killing some of them to break the cycles.
What to check: Find the conflicting statements in the PostgreSQL log (deadlock reports name both queries) and address the access pattern that lets them collide.
PostgresTxDurationTooLong⚓︎
| Severity | Fires after |
|---|---|
| Warning | Immediately |
Trigger condition
pg_stat_activity_max_tx_duration{application_name!="pg_dump"} > 800
A transaction (backups via pg_dump excluded) has been open for more than 800 seconds. Long-running transactions hold locks and prevent vacuum from cleaning up, degrading the whole database over time.
What to check: Find the transaction in pg_stat_activity, identify its client, and terminate it if it is stuck. Investigate why the client left it open.
PostgresClusterOutOfSync⚓︎
| Severity | Fires after |
|---|---|
| Critical | 2m |
Trigger condition
pg_replication_slots_pg_wal_lsn_diff > 500000000
A replica is lagging the primary by more than 500 MB of WAL. A failover while lagging this far would lose data, and unbounded lag can fill the primary's disk with retained WAL.
What to check: Check that the replica is running and applying WAL, and that the network between primary and replica is healthy. If the replica is permanently gone, drop its replication slot so the primary stops retaining WAL for it.
PostgresCollectorUnhealthy⚓︎
| Severity | Fires after |
|---|---|
| Critical | 1m |
Trigger condition
up{job="postgres",site_state!~"decommissioned|maintenance"} == 0 or pg_exporter_last_scrape_error{job="postgres",site_state!~"decommissioned|maintenance"} == 1
The PostgreSQL metrics exporter is down or cannot probe the database. While it is unhealthy, all the database alerts above are blind.
What to check: Check the exporter container, then its connection to the database (credentials, reachability). If the database itself is down, that is the real incident.