Platform and Infrastructure Alerts⚓︎

This page is the reference for the default alerts that apply to every Expeto deployment regardless of component: the health of the Kubernetes nodes everything runs on, application log signals, liveness of nodes and of the monitoring pipeline itself, and the optional metric export integrations. Alerts specific to one component are on their own pages: xCore alerts and xControl alerts. The endpoints these alerts draw their metrics from are listed in Monitored Endpoints.

How to read these alerts⚓︎

Each alert entry on this page — and on the xCore and xControl alert pages — shows the alert's severity, how long its trigger condition must hold before it fires ("Fires after"), the exact PromQL trigger expression as deployed, and a short guide to what the alert means and what to check. How alerts are defined, and how some alerts escalate through Tier2/Tier3 variants as a condition persists, is explained in How alerts work.

Severity reflects the intended response:

Firing alerts carry labels identifying the affected customer, site and node, which the notification templates use; filter by these labels in Grafana or the metric database to find all alerts for one site or customer.

Alert summary⚓︎

Node health⚓︎

These alerts come from node_exporter metrics collected on every Kubernetes node (the node scrape job). The checks below are run on the node itself — over SSH where the cluster allows it, or with kubectl debug node/<name> for a shell with access to the node.

DiskInReadOnlyMode⚓︎

Trigger condition

node_filesystem_readonly{site_state!~"decommissioned|maintenance",fstype!="squashfs"} == 1

A filesystem on the affected node has been remounted read-only, which the kernel typically does after underlying storage errors. Every service writing to that filesystem is failing.

What to check: Inspect the kernel log (dmesg) on the node for the storage error that triggered the remount, and check the health of the underlying disk or storage backend. Repair or replace the storage before remounting read-write; a reboot alone usually only postpones the problem.

DiskIOUnhealthy⚓︎

Trigger condition

increase(node_disk_write_time_seconds_total[1m])>500

Writes on a device are spending an excessive amount of time in flight — the disk is saturated or degrading.

What to check: Look at I/O statistics on the node (iostat -x) to see whether a workload is saturating the device or the device itself is slow. Databases on the node are usually the first services to suffer.

DiskSpaceLow⚓︎

Trigger condition

100 - ((node_filesystem_avail_bytes{site_state!~"decommissioned|maintenance",fstype!="squashfs"} * 100) / node_filesystem_size_bytes{site_state!~"decommissioned|maintenance",fstype!="squashfs"}) > 90

Disk usage on a filesystem has exceeded 90 percent on the affected node. A full disk halts databases, the site collector, and every pod scheduled on the node.

What to check: Identify what is consuming space on the reported device (df -h, then look for large log files, core dumps, or old images) and free space or expand the volume.

DiskSpaceFilling⚓︎

Trigger condition

(node_filesystem_avail_bytes{job="node",fstype!="",site_state!~"decommissioned|maintenance"} / node_filesystem_size_bytes{job="node",fstype!="",site_state!~"decommissioned|maintenance"} * 100 < 40 and  predict_linear(node_filesystem_avail_bytes{job="node",fstype!="",site_state!~"decommissioned|maintenance"}[6h], 24*60*60) < 0  and node_filesystem_readonly{job="node",fstype!="",site_state!~"decommissioned|maintenance"} == 0 )

A filesystem is below 40 percent free and, at its current growth rate over the last 6 hours, is predicted to fill completely within 24 hours. This is the early warning before DiskSpaceLow.

What to check: Find what is growing (usually logs or database data) and address it before the disk actually fills. If the growth is legitimate, plan a volume expansion now.

HostHighCpuLoad⚓︎

Trigger condition

100 - max without (cpu) ( max without (mode) (rate(node_cpu_seconds_total{mode="idle",site_state!~"decommissioned|maintenance"}[5m])) + max without (mode) (rate(node_cpu_seconds_total{mode="nice",site_state!~"decommissioned|maintenance"}[5m])) ) * 100 > 85

CPU load on the affected node has averaged above 85 percent for 5 minutes.

What to check: Identify the busy pods (kubectl top pods --all-namespaces --sort-by=cpu) or processes on the node (top). A one-off spike may be a batch job; sustained saturation means the node is undersized for its workload or a process is misbehaving.

HostHighMemoryUsage⚓︎

Trigger condition

100-(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes{site_state!~"decommissioned|maintenance"} * 100 ) > 90

Memory usage on the affected node has been above 90 percent for 5 minutes. If it keeps climbing, the kubelet starts evicting pods and the kernel starts killing processes.

What to check: Identify the largest consumers (kubectl top pods --all-namespaces --sort-by=memory) and whether one of them is leaking. If usage is legitimately high, the node needs more memory.

HostClockNotSynchronising⚓︎

Trigger condition

min_over_time(node_timex_sync_status{site_state!~"decommissioned|maintenance"}[1h]) == 0 and node_timex_maxerror_seconds{site_state!~"decommissioned|maintenance"} >= 16

The node's clock has not synchronized with NTP for the past hour and its estimated error is at least 16 seconds. Clock drift breaks certificate validation, authentication, and log correlation.

What to check: Check the time service on the node (chronyc tracking or timedatectl) and whether the configured NTP servers are reachable from the site.

NetworkInterfaceDown⚓︎

Trigger condition

node_network_up{site_state!~"decommissioned|maintenance",device=~"eth.*"} == 0

A physical network interface (eth*) on the affected node has been down for 5 minutes.

What to check: Check the interface state on the node (ip link), then the layer below it — cabling or switch port on bare metal, the virtual NIC on a hypervisor.

Error logs⚓︎

These alerts are derived from log-level counters exposed by the Expeto Java applications, so they fire on what the applications themselves report rather than on resource metrics.

ErrorLogs⚓︎

Trigger condition

increase(log_error_total{site_state!~"decommissioned|maintenance",logger!~"io.expeto.thunder.rest.error.RestExceptionHandler"}[1m]) > 1

An Expeto component has logged more than one error-level message in the last minute. A few known-noisy loggers are excluded from the expression; the deployed rule also excludes loggers belonging to retired subsystems, which are omitted here because the code they name no longer exists.

What to check: Open the logs of the component named in the alert's labels around the time the alert fired and read the actual error. This alert is a pointer, not a diagnosis.

FatalLogs⚓︎

Trigger condition

increase(log_fatal_total{site_state!~"decommissioned|maintenance"}[1m]) > 1

An Expeto component has logged fatal-level messages — it has most likely crashed or aborted an operation it could not recover from.

What to check: Go straight to the component's logs, then check whether its pod is still running and whether it has restarted (see KubePodCrashLooping). Treat the service as impaired until proven otherwise.

JVMThreadsTooHigh⚓︎

Trigger condition

jvm_threads_current > 500

An Expeto JVM is running more than 500 threads — usually a thread leak or a processing backlog rather than legitimate load.

What to check: Identify the component from the alert labels and take a thread dump to see what the threads are doing. A restart clears the symptom but the dump tells you the cause.

Basic up and down⚓︎

Liveness alerts: nodes, the site collectors, and the monitoring pipeline itself.

NodeDown⚓︎

Trigger condition

up{job="node",site_state!~"decommissioned|maintenance"} == 0

A node's node_exporter is no longer reachable — the node is down, or it is up but not reporting any metrics.

What to check: Determine whether the whole node is down or only its exporter: kubectl get nodes shows the node's readiness, and if its kubelet and cAdvisor targets are failing at the same time the node itself is gone. A dead node is an infrastructure incident; a dead exporter is a monitoring gap to fix promptly.

PrometheusCollectorDown⚓︎

Trigger condition

up{job="prometheus",site_state!~"decommissioned|maintenance"} == 0

A Prometheus instance (typically a site collector) is down. Metrics from everything it scrapes are not being collected, and since collectors retain only a short history, an extended outage loses data permanently.

What to check: Check the collector's pod in the site's namespace — kubectl get pods, then its logs and events — and restore it promptly.

PrometheusFederationDown⚓︎

Trigger condition

up{job=~".*federate.*",site_state!~"decommissioned|maintenance"} == 0

A federation scrape has failed: xView can no longer pull metrics directly from one of the collectors it federates from (see central xView scrape jobs). Alerting for everything behind that collector is degraded until federation is restored. This alert applies only where a federation scrape job is configured; the standard deployment relays every site through xControl and does not federate, so in that case there are no targets for it to evaluate.

What to check: Check network reachability from xView to the collector named in the failing job, then the collector's own health, and the credentials/TLS used by the federation job.

PrometheusScrapeDurationHigh⚓︎

Trigger condition

scrape_duration_seconds{site_state!~"decommissioned|maintenance"} > 8

Scraping a target has been taking more than 8 seconds — dangerously close to the 10-second scrape timeout. If the timeout is hit, samples are dropped and gaps appear in the data.

What to check: Check the load on the target named in the alert and the network path to it. A target exposing an unusually large number of metrics can also push scrape time up.

Kafka remote write⚓︎

Health of the Kafka export adapter, which forwards platform metrics to an external Kafka topic (see exporting metrics onward).

KafkaRemoteWriteErrors⚓︎

Trigger condition

increase(objects_failed_total[5m])>0 OR increase(serialized_failed_total[5m])>0

The Kafka remote-write adapter has failed to serialize or deliver metric objects in the last 5 minutes. The external consumer of these metrics is missing data.

What to check: Check the adapter's logs for the failure reason, and the availability of the downstream Kafka brokers and topic.

Splunk remote write⚓︎

Health of the Splunk export adapter (see exporting metrics onward).

SplunkRemoteWriteErrors⚓︎

Trigger condition

increase(ropee_splunk_events_wrote_failed_count[5m])>0

The Splunk remote-write adapter has failed to write events in the last 5 minutes. Metrics destined for Splunk are being lost.

What to check: Check the adapter's logs for the failure reason, and the availability of the downstream Splunk endpoint it writes to.