Skip to content

Infrastructure Metrics⚓︎

Alongside the platform's own metrics, xView collects the standard measurements of the machines and Kubernetes clusters everything runs on. These come from well-known open-source exporters, and this page is a guide to which exporter covers what, with the queries worth knowing — not a restatement of their reference documentation, which is linked and is authoritative.

The alerts built on these metrics are documented in Platform and Infrastructure Alerts and Kubernetes and Monitoring Stack Alerts.

Host metrics⚓︎

Host-level measurements come from node_exporter, running on every cluster node, under the node job. The full metric set is large and documented upstream at prometheus/node_exporter.

The families that matter most in practice:

Family Covers
node_cpu_seconds_total CPU time, by core and mode. A counter — always use rate().
node_memory_MemTotal_bytes, node_memory_MemAvailable_bytes Memory. Prefer available over free: free excludes reclaimable cache and will look alarmingly low on a healthy machine.
node_filesystem_size_bytes, node_filesystem_avail_bytes Filesystem capacity and free space, per mount
node_filesystem_readonly Whether a filesystem has been remounted read-only, which the kernel does after storage errors
node_disk_read_bytes_total, node_disk_write_bytes_total, node_disk_io_time_seconds_total Disk throughput and utilization
node_network_receive_bytes_total, node_network_transmit_bytes_total Network throughput, per interface
node_network_up Interface link state
node_load1, node_load5, node_load15 Load average
node_time_seconds, node_timex_sync_status Clock and time synchronization
node_boot_time_seconds Boot time, from which uptime is derived

Queries worth knowing⚓︎

CPU utilization as a percentage, per node:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Memory utilization, using available rather than free:

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Filesystem usage, excluding the pseudo-filesystems that would otherwise dominate the result:

100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|squashfs|overlay"}
           / node_filesystem_size_bytes{fstype!~"tmpfs|squashfs|overlay"})

Projected time until a filesystem fills, which is a far better alerting signal than a fixed threshold — it distinguishes a disk sitting steadily at 85 percent from one that will be full by morning:

predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|squashfs|overlay"}[6h], 24 * 3600) < 0

Note

Clock synchronization is worth checking early when timestamps look wrong, metrics arrive out of order, or authentication fails intermittently. Time skew produces symptoms that look like unrelated faults — see HostClockNotSynchronising.

Container metrics⚓︎

Per-container resource usage comes from cAdvisor, which is built into the kubelet, under the kubelet job. Reference documentation is at google/cadvisor.

Family Covers
container_cpu_usage_seconds_total CPU time consumed by a container
container_memory_working_set_bytes Memory in use. This is the figure the kernel compares against the limit when deciding to kill a container.
container_memory_rss, container_memory_cache Memory breakdown
container_fs_usage_bytes, container_fs_limit_bytes Container filesystem usage
container_network_receive_bytes_total, container_network_transmit_bytes_total Per-container network traffic, by interface

Important

Use container_memory_working_set_bytes, not container_memory_usage_bytes, when comparing against a memory limit. The latter includes reclaimable page cache, so it overstates real usage and will suggest a container is near its limit when it is not.

Memory headroom for a container, as a proportion of its limit:

container_memory_working_set_bytes{container!=""}
  / on (container, pod, namespace)
kube_pod_container_resource_limits{resource="memory"}

Kubernetes object state⚓︎

kube-state-metrics reports the state of Kubernetes objects themselves — not their resource usage, but what the API server believes exists and whether it is healthy. It runs under the k8s job, and its metrics are documented per object type at kube-state-metrics.

Family Covers
kube_pod_status_phase Pod phase: pending, running, succeeded, failed
kube_pod_container_status_restarts_total Container restarts, the signal for a crash loop
kube_pod_container_status_waiting_reason Why a container is not running, including image-pull failures
kube_deployment_status_replicas_available vs kube_deployment_spec_replicas Whether a workload is fully scheduled
kube_node_status_condition Node readiness, memory and disk pressure
kube_persistentvolumeclaim_status_phase Whether storage is bound
kube_pod_container_resource_requests, ..._limits Configured requests and limits, for comparison against actual usage

Workloads that are not fully available:

kube_deployment_spec_replicas - kube_deployment_status_replicas_available > 0

Containers restarting repeatedly:

increase(kube_pod_container_status_restarts_total[1h]) > 3

Kubernetes control plane⚓︎

The API server, scheduler, controller manager, etcd and CoreDNS each expose their own metrics, and xView scrapes them where they are reachable. On a managed Kubernetes service the control plane is usually not exposed, and those targets will be absent — that is expected, not a fault.

The corresponding alerts are documented in Kubernetes and Monitoring Stack Alerts, which is generated from the alert definitions themselves and includes each alert's upstream runbook link.

A note on where to look first⚓︎

When a site is in trouble, these metrics answer a different question from the platform metrics, and the order matters. Platform metrics tell you what is failing — attach is failing, sessions are not establishing. Infrastructure metrics tell you why it might be — the node is out of memory, the disk is full, the container is being restarted.

Start with the platform metrics to establish the symptom, then use these to explain it. Going the other way around tends to produce plausible-looking infrastructure findings that turn out to be unrelated.