Skip to content

xControl Metrics⚓︎

This page is the reference for the metrics xControl reports about itself: the work queues that carry configuration out to the sites, the caches and thread pools that serve the API, the database behind it, and the Java runtime it runs on.

These metrics are scraped directly from xControl under the job name xControl. Unlike site metrics, they carry no site or customer labels — xControl is a single central service, not something that exists per site. The alerts built on them are documented in xControl Alerts.

Message queues⚓︎

xControl communicates with each site's agent asynchronously. Configuration changes are placed on a queue and delivered when the site is reachable, which is what allows a site to be offline without losing changes.

Metric Meaning
xcontrol_queue_size Messages currently waiting on a queue
xcontrol_queue_total Messages processed
xcontrol_queue_health Health of the queue subsystem
xcontrol_jms_event_time, xcontrol_jms_event_time_count Time taken to process an event, and the number processed
xcontrol_jms_delay_time, xcontrol_jms_delay_time_count Time a message waited before being processed

Queue depth is the number to watch. A queue that is growing rather than draining means configuration changes are not reaching sites, and subscribers provisioned in the API will not work on the network until it clears. This is what JMSQueueNotDraining detects.

xcontrol_queue_size

Rising delay time with a steady queue size is a different problem — messages are being processed, but slowly. Check the database and thread pools before assuming a site is unreachable.

Delivery outcomes are counted by expeto_jms_event_success_total and expeto_jms_event_failure_total, which the Expeto Agent also reports for its own side of the same exchange.

Thread pools⚓︎

xControl separates fast operations from slow ones so that a long-running request cannot starve the interactive API.

Metric Meaning
xcontrol_threadpool_fast, xcontrol_threadpool_fast_count The fast pool's utilization and task count
xcontrol_threadpool_slow, xcontrol_threadpool_slow_count The slow pool's utilization and task count

A saturated fast pool shows up to users as a slow interface, and usually indicates work that belongs on the slow pool, or a downstream dependency that has become slow.

Caches⚓︎

Metric Meaning
xcontrol_cache_count Entries currently cached
xcontrol_cache_total Cache capacity
xcontrol_cache_utilization Proportion of capacity in use

Sustained utilization at capacity means entries are being evicted and re-fetched, which turns cache reads into database reads.

External service credits⚓︎

Some features call metered third-party services. These gauges report the balance remaining, so the service can be topped up before it stops working:

Metric Service
xcontrol_imeilookup_credits Device identity lookups
xcontrol_cpsapi_credits Location lookups

Both drive alerts — ImeiLookupCreditsLow and LocationLookupApiCreditsLow — because the failure mode when they run out is a feature that silently stops returning results.

Subscribers and tokens⚓︎

Metric Meaning
xcontrol_subscribers_active30d Subscribers active in the last 30 days
xcontrol_token_expiration Time remaining before an API token expires

xcontrol_token_expiration is what SiteApiTokenExpiringSoon evaluates. An expired site token stops that site's agent from authenticating, which presents as the site going out of contact.

Database⚓︎

xControl stores its data in PostgreSQL. Its own view of the database:

Metric Meaning
xcontrol_postgres_preparedtransactionsage Age of the oldest prepared transaction

An ageing prepared transaction blocks vacuum and will eventually degrade the whole database, which is why it is monitored separately from ordinary transaction duration.

Connection-pool health is reported by the pool itself, under the HikariPool_* family:

Metric Meaning
HikariPool_1_pool_ActiveConnections Connections currently in use
HikariPool_1_pool_IdleConnections Connections available
HikariPool_1_pool_TotalConnections Connections open
HikariPool_1_pool_MaxConnections, HikariPool_1_pool_MinConnections Configured bounds
HikariPool_1_pool_PendingConnections Threads waiting for a connection
HikariPool_1_pool_Wait, HikariPool_1_pool_Wait_count Time spent waiting for a connection
HikariPool_1_pool_Usage, HikariPool_1_pool_Usage_count How long connections are held
HikariPool_1_pool_ConnectionTimeoutRate_total Requests that gave up waiting

Pending connections above zero for any sustained period means the pool is too small for the load, or queries are holding connections too long.

The database server itself is monitored separately by postgres_exporter under the postgres job, producing the standard pg_* metrics — see postgres_exporter for the full set, and xControl Alerts for the alerts built on them.

Java runtime⚓︎

xControl is a JVM application and exports the standard runtime metrics defined by the Prometheus Java client:

Family Covers
jvm_memory_bytes_used, jvm_memory_bytes_committed, jvm_memory_bytes_max Heap and non-heap memory, by area
jvm_memory_pool_bytes_* The same, per memory pool
jvm_gc_collection_seconds_* Garbage collection count and time, by collector
jvm_threads_current, jvm_threads_peak, jvm_threads_deadlocked Thread counts
jvm_classes_currently_loaded, jvm_classes_loaded_total Class loading
jvm_buffer_pool_* Direct and mapped buffer usage
process_cpu_seconds_total, process_resident_memory_bytes, process_open_fds Process-level resources

Thread count is the one to watch: JVMThreadsTooHigh fires on it, because runaway thread creation usually precedes a failure rather than following one.

Logging⚓︎

log_error_total and log_fatal_total count log events by level, labelled by logger, and drive ErrorLogs and FatalLogs. The logger label makes it possible to exclude a known-noisy source from an alert without losing the signal elsewhere.