Kubernetes and Monitoring Stack Alerts⚓︎

xView runs its Kubernetes monitoring on the victoria-metrics-k8s-stack Helm chart, which ships a large body of alerting rules of its own: the Kubernetes and node_exporter mixins, the etcd mixin, and VictoriaMetrics' rules for its own components. This page is the reference for those alerts. They are evaluated and routed exactly like Expeto's own rules, which are documented separately on Platform and Infrastructure Alerts, xCore Alerts, and xControl Alerts.

How to read these alerts⚓︎

Each entry gives the alert's severity, how long its condition must hold before it fires, the trigger expression as deployed, the upstream description, and — where upstream provides one — a link to its runbook. The {{ ... }} placeholders in a description are substituted with the firing series' own labels when the notification is delivered.

Severity means the same thing here as on the Expeto alert pages:

Some alerts are defined more than once under a single name — usually as a warning and a critical variant of the same condition at different thresholds, and in one case as four error-budget burn rate windows. Those appear as one entry with a row per variant.

Alerts operators most often ask about⚓︎

Operators integrating against the platform ask about the alerts below more than any others. It is a starting point for triage, not a statement of which alerts you should route or page on; the full reference follows.

Alert summary⚓︎

General⚓︎

Chart-wide signals rather than alerts about any one component. TargetDown reports that a meaningful share of a scrape job's targets are unreachable. The other two never reach an operator: Watchdog fires permanently so that a dead alerting pipeline can be detected by its absence, and InfoInhibitor exists so Alertmanager can suppress info-level noise while a real alert is already firing for the same namespace.

TargetDown⚓︎

One or more targets are unreachable.

{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace are down.

Trigger condition

(100 * (count(up == 0) by(cluster,job,namespace,service) / count(up) by(cluster,job,namespace,service))) > 10

Upstream runbook for TargetDown

Watchdog⚓︎

An alert that should always be firing to certify that Alertmanager is working properly.

This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always firing, therefore it should always be firing in Alertmanager and always fire against a receiver. There are integrations with various notification mechanisms that send a notification when this alert is not firing. For example the "DeadMansSnitch" integration in PagerDuty.

Trigger condition

vector(1)

Upstream runbook for Watchdog

InfoInhibitor⚓︎

Info-level alert inhibition.

This is an alert that is used to inhibit info alerts. By themselves, the info-level alerts are sometimes very noisy, but they are relevant when combined with other alerts. This alert fires whenever there's a severity="info" alert, and stops firing when another alert with a severity of 'warning' or 'critical' starts firing on the same namespace. This alert should be routed to a null receiver and configured to inhibit alerts with severity="info".

Trigger condition

group(ALERTS{severity="info"} == 1) by(namespace,cluster) unless on(namespace,cluster) group(ALERTS{alertname!="InfoInhibitor",alertstate="firing",severity=~"warning|critical"} == 1) by(namespace,cluster)

Upstream runbook for InfoInhibitor

Kubernetes workloads⚓︎

Workload-level alerts derived from kube-state-metrics: Deployments, StatefulSets, DaemonSets, Jobs, HorizontalPodAutoscalers, PodDisruptionBudgets, and individual Pods. Most wait 15 minutes before firing, which is long enough to ride out a normal rollout.

KubePodCrashLooping⚓︎

Pod is crash looping.

Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: "CrashLoopBackOff") on cluster {{ $labels.cluster }}.

Trigger condition

max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff",namespace=~".*",job="kube-state-metrics"}[5m]) >= 1

Upstream runbook for KubePodCrashLooping

KubePodNotReady⚓︎

Pod has been in a non-ready state for more than 15 minutes.

Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

sum(max(kube_pod_status_phase{namespace=~".*",job="kube-state-metrics",phase=~"Pending|Unknown"}) by(namespace,pod,job,cluster) * on(namespace,pod,cluster) group_left() topk(1, max(kube_pod_owner{owner_kind!="Job"}) by(namespace,pod,owner_kind,cluster)) by(namespace,pod,cluster)) by(namespace,pod,job,cluster) > 0

Upstream runbook for KubePodNotReady

KubeDeploymentGenerationMismatch⚓︎

Deployment generation mismatch due to possible roll-back

Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back on cluster {{ $labels.cluster }}.

Trigger condition

kube_deployment_status_observed_generation{namespace=~".*",job="kube-state-metrics"} != kube_deployment_metadata_generation{namespace=~".*",job="kube-state-metrics"}

Upstream runbook for KubeDeploymentGenerationMismatch

KubeDeploymentReplicasMismatch⚓︎

Deployment has not matched the expected number of replicas.

Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

(kube_deployment_spec_replicas{namespace=~".*",job="kube-state-metrics"} > kube_deployment_status_replicas_available{namespace=~".*",job="kube-state-metrics"}) and (changes(kube_deployment_status_replicas_updated{namespace=~".*",job="kube-state-metrics"}[10m]) == 0)

Upstream runbook for KubeDeploymentReplicasMismatch

KubeDeploymentRolloutStuck⚓︎

Deployment rollout is not progressing.

Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} is not progressing for longer than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

kube_deployment_status_condition{condition="Progressing",status="false",namespace=~".*",job="kube-state-metrics"} != 0

Upstream runbook for KubeDeploymentRolloutStuck

KubeStatefulSetReplicasMismatch⚓︎

StatefulSet has not matched the expected number of replicas.

StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

(kube_statefulset_status_replicas_ready{namespace=~".*",job="kube-state-metrics"} != kube_statefulset_replicas{namespace=~".*",job="kube-state-metrics"}) and (changes(kube_statefulset_status_replicas_updated{namespace=~".*",job="kube-state-metrics"}[10m]) == 0)

Upstream runbook for KubeStatefulSetReplicasMismatch

KubeStatefulSetGenerationMismatch⚓︎

StatefulSet generation mismatch due to possible roll-back

StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back on cluster {{ $labels.cluster }}.

Trigger condition

kube_statefulset_status_observed_generation{namespace=~".*",job="kube-state-metrics"} != kube_statefulset_metadata_generation{namespace=~".*",job="kube-state-metrics"}

Upstream runbook for KubeStatefulSetGenerationMismatch

KubeStatefulSetUpdateNotRolledOut⚓︎

StatefulSet update has not been rolled out.

StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out on cluster {{ $labels.cluster }}.

Trigger condition

(max(kube_statefulset_status_current_revision{namespace=~".*",job="kube-state-metrics"} unless kube_statefulset_status_update_revision{namespace=~".*",job="kube-state-metrics"}) by(namespace,statefulset,job,cluster) * on(namespace,statefulset,job,cluster) (kube_statefulset_replicas{namespace=~".*",job="kube-state-metrics"} != kube_statefulset_status_replicas_updated{namespace=~".*",job="kube-state-metrics"})) and on(namespace,statefulset,job,cluster) (changes(kube_statefulset_status_replicas_updated{namespace=~".*",job="kube-state-metrics"}[5m]) == 0)

Upstream runbook for KubeStatefulSetUpdateNotRolledOut

KubeDaemonSetRolloutStuck⚓︎

DaemonSet rollout is stuck.

DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 15m on cluster {{ $labels.cluster }}.

Trigger condition

((((kube_daemonset_status_current_number_scheduled{namespace=~".*",job="kube-state-metrics"} != kube_daemonset_status_desired_number_scheduled{namespace=~".*",job="kube-state-metrics"}) or (kube_daemonset_status_number_misscheduled{namespace=~".*",job="kube-state-metrics"} != 0)) or (kube_daemonset_status_updated_number_scheduled{namespace=~".*",job="kube-state-metrics"} != kube_daemonset_status_desired_number_scheduled{namespace=~".*",job="kube-state-metrics"})) or (kube_daemonset_status_number_available{namespace=~".*",job="kube-state-metrics"} != kube_daemonset_status_desired_number_scheduled{namespace=~".*",job="kube-state-metrics"})) and (changes(kube_daemonset_status_updated_number_scheduled{namespace=~".*",job="kube-state-metrics"}[5m]) == 0)

Upstream runbook for KubeDaemonSetRolloutStuck

KubeContainerWaiting⚓︎

Pod container waiting longer than 1 hour

pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour. (reason: "{{ $labels.reason }}") on cluster {{ $labels.cluster }}.

Trigger condition

kube_pod_container_status_waiting_reason{reason!="CrashLoopBackOff",namespace=~".*",job="kube-state-metrics"} > 0

Upstream runbook for KubeContainerWaiting

KubeDaemonSetNotScheduled⚓︎

DaemonSet pods are not scheduled.

{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled on cluster {{ $labels.cluster }}.

Trigger condition

(kube_daemonset_status_desired_number_scheduled{namespace=~".*",job="kube-state-metrics"} - kube_daemonset_status_current_number_scheduled{namespace=~".*",job="kube-state-metrics"}) > 0

Upstream runbook for KubeDaemonSetNotScheduled

KubeDaemonSetMisScheduled⚓︎

DaemonSet pods are misscheduled.

{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run on cluster {{ $labels.cluster }}.

Trigger condition

kube_daemonset_status_number_misscheduled{namespace=~".*",job="kube-state-metrics"} > 0

Upstream runbook for KubeDaemonSetMisScheduled

KubeJobNotCompleted⚓︎

Job did not complete in time

Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than {{ "43200" | humanizeDuration }} to complete on cluster {{ $labels.cluster }}.

Trigger condition

(time() - max(kube_job_status_start_time{namespace=~".*",job="kube-state-metrics"} and (kube_job_status_active{namespace=~".*",job="kube-state-metrics"} > 0)) by(namespace,job_name,cluster)) > 43200

Upstream runbook for KubeJobNotCompleted

KubeJobFailed⚓︎

Job failed to complete.

Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert on cluster {{ $labels.cluster }}.

Trigger condition

kube_job_failed{namespace=~".*",job="kube-state-metrics"} > 0

Upstream runbook for KubeJobFailed

KubeHpaReplicasMismatch⚓︎

HPA has not matched desired number of replicas.

HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} has not matched the desired number of replicas for longer than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

(((kube_horizontalpodautoscaler_status_desired_replicas{namespace=~".*",job="kube-state-metrics"} != kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"}) and (kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"} > kube_horizontalpodautoscaler_spec_min_replicas{namespace=~".*",job="kube-state-metrics"})) and (kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"} < kube_horizontalpodautoscaler_spec_max_replicas{namespace=~".*",job="kube-state-metrics"})) and (changes(kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"}[15m]) == 0)

Upstream runbook for KubeHpaReplicasMismatch

KubeHpaMaxedOut⚓︎

HPA is running at max replicas

HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} has been running at max replicas for longer than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

(kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"} == kube_horizontalpodautoscaler_spec_max_replicas{namespace=~".*",job="kube-state-metrics"}) and on(namespace,horizontalpodautoscaler,cluster) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~".*",job="kube-state-metrics"} != kube_horizontalpodautoscaler_spec_min_replicas{namespace=~".*",job="kube-state-metrics"})

Upstream runbook for KubeHpaMaxedOut

KubePdbNotEnoughHealthyPods⚓︎

PDB does not have enough healthy pods.

PDB {{ $labels.cluster }}/{{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} expects {{ $value }} more healthy pods. The desired number of healthy pods has not been met for at least 15m.

Trigger condition

(kube_poddisruptionbudget_status_desired_healthy{namespace=~".*",job="kube-state-metrics"} - kube_poddisruptionbudget_status_current_healthy{namespace=~".*",job="kube-state-metrics"}) > 0

Upstream runbook for KubePdbNotEnoughHealthyPods

Kubernetes resource usage⚓︎

Pod requests, limits, and namespace quotas measured against what the cluster can actually schedule. These describe capacity planning rather than a live outage — an overcommitted cluster runs fine until a node is lost.

KubeCPUOvercommit⚓︎

Cluster has overcommitted CPU resource requests.

Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Pods by {{ printf "%.2f" $value }} CPU shares and cannot tolerate node failure.

Trigger condition

(((sum(namespace_cpu:kube_pod_container_resource_requests:sum) by(cluster) - sum(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by(cluster)) > 0) and (count(max(kube_node_role{job="kube-state-metrics",role="control-plane"}) by(cluster,node)) by(cluster) < 3)) or ((sum(namespace_cpu:kube_pod_container_resource_requests:sum) by(cluster) - ((sum(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by(cluster) - max(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by(cluster)) > 0)) > 0)

Upstream runbook for KubeCPUOvercommit

KubeMemoryOvercommit⚓︎

Cluster has overcommitted memory resource requests.

Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.

Trigger condition

(((sum(namespace_memory:kube_pod_container_resource_requests:sum) by(cluster) - sum(kube_node_status_allocatable{job="kube-state-metrics",resource="memory"}) by(cluster)) > 0) and (count(max(kube_node_role{job="kube-state-metrics",role="control-plane"}) by(cluster,node)) by(cluster) < 3)) or ((sum(namespace_memory:kube_pod_container_resource_requests:sum) by(cluster) - ((sum(kube_node_status_allocatable{job="kube-state-metrics",resource="memory"}) by(cluster) - max(kube_node_status_allocatable{job="kube-state-metrics",resource="memory"}) by(cluster)) > 0)) > 0)

Upstream runbook for KubeMemoryOvercommit

KubeCPUQuotaOvercommit⚓︎

Cluster has overcommitted CPU resource requests.

Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Namespaces.

Trigger condition

(sum(min(kube_resourcequota{job="kube-state-metrics",type="hard",resource=~"(cpu|requests.cpu)"}) without(resource)) by(cluster) / sum(kube_node_status_allocatable{resource="cpu",job="kube-state-metrics"}) by(cluster)) > 1.5

Upstream runbook for KubeCPUQuotaOvercommit

KubeMemoryQuotaOvercommit⚓︎

Cluster has overcommitted memory resource requests.

Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Namespaces.

Trigger condition

(sum(min(kube_resourcequota{job="kube-state-metrics",type="hard",resource=~"(memory|requests.memory)"}) without(resource)) by(cluster) / sum(kube_node_status_allocatable{resource="memory",job="kube-state-metrics"}) by(cluster)) > 1.5

Upstream runbook for KubeMemoryQuotaOvercommit

KubeQuotaAlmostFull⚓︎

Namespace quota is going to be full.

Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota on cluster {{ $labels.cluster }}.

Trigger condition

((max(kube_resourcequota{job="kube-state-metrics",type="used"}) without(instance,job,type) / on(cluster,namespace,resource,resourcequota) group_left() (max(kube_resourcequota{job="kube-state-metrics",type="hard"}) without(instance,job,type) > 0)) > 0.9) < 1

Upstream runbook for KubeQuotaAlmostFull

KubeQuotaFullyUsed⚓︎

Namespace quota is fully used.

Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota on cluster {{ $labels.cluster }}.

Trigger condition

(max(kube_resourcequota{job="kube-state-metrics",type="used"}) without(instance,job,type) / on(cluster,namespace,resource,resourcequota) group_left() (max(kube_resourcequota{job="kube-state-metrics",type="hard"}) without(instance,job,type) > 0)) == 1

Upstream runbook for KubeQuotaFullyUsed

KubeQuotaExceeded⚓︎

Namespace quota has exceeded the limits.

Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota on cluster {{ $labels.cluster }}.

Trigger condition

(max(kube_resourcequota{job="kube-state-metrics",type="used"}) without(instance,job,type) / on(cluster,namespace,resource,resourcequota) group_left() (max(kube_resourcequota{job="kube-state-metrics",type="hard"}) without(instance,job,type) > 0)) > 1

Upstream runbook for KubeQuotaExceeded

CPUThrottlingHigh⚓︎

Processes experience elevated CPU throttling.

{{ $value | humanizePercentage }} throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod }} on cluster {{ $labels.cluster }}.

Trigger condition

(sum(topk(1, increase(container_cpu_cfs_throttled_periods_total{container!="",job="kubelet",metrics_path="/metrics/cadvisor"}[5m])) by(cluster,namespace,pod,container,instance)) without(id,metrics_path,name,image,endpoint,job,node) / on(cluster,namespace,pod,container,instance) group_left() sum(topk(1, increase(container_cpu_cfs_periods_total{job="kubelet",metrics_path="/metrics/cadvisor"}[5m])) by(cluster,namespace,pod,container,instance)) without(id,metrics_path,name,image,endpoint,job,node)) > 0.25

Upstream runbook for CPUThrottlingHigh

Kubernetes storage⚓︎

PersistentVolume capacity, inode exhaustion, and provisioning failures. The two filling-up alerts combine a current threshold with a predict_linear forecast, so they fire before the volume is actually full.

KubePersistentVolumeFillingUp⚓︎

PersistentVolume is filling up.

This alert is defined in 2 variants:

Critical, fires after 1m

The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} on Cluster {{ $labels.cluster }} is only {{ $value | humanizePercentage }} free.

((((kubelet_volume_stats_available_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"} / kubelet_volume_stats_capacity_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"}) < 0.03) and (kubelet_volume_stats_used_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"} > 0)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1)

Warning, fires after 1h

Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} on Cluster {{ $labels.cluster }} is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available.

(((((kubelet_volume_stats_available_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"} / kubelet_volume_stats_capacity_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"}) < 0.15) and (kubelet_volume_stats_used_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"} > 0)) and (predict_linear(kubelet_volume_stats_available_bytes{namespace=~".*",job="kubelet",metrics_path="/metrics"}[6h], 345600) < 0)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1)

Upstream runbook for KubePersistentVolumeFillingUp

KubePersistentVolumeInodesFillingUp⚓︎

PersistentVolumeInodes are filling up.

This alert is defined in 2 variants:

Critical, fires after 1m

The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} on Cluster {{ $labels.cluster }} only has {{ $value | humanizePercentage }} free inodes.

((((kubelet_volume_stats_inodes_free{namespace=~".*",job="kubelet",metrics_path="/metrics"} / kubelet_volume_stats_inodes{namespace=~".*",job="kubelet",metrics_path="/metrics"}) < 0.03) and (kubelet_volume_stats_inodes_used{namespace=~".*",job="kubelet",metrics_path="/metrics"} > 0)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1)

Warning, fires after 1h

Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} on Cluster {{ $labels.cluster }} is expected to run out of inodes within four days. Currently {{ $value | humanizePercentage }} of its inodes are free.

(((((kubelet_volume_stats_inodes_free{namespace=~".*",job="kubelet",metrics_path="/metrics"} / kubelet_volume_stats_inodes{namespace=~".*",job="kubelet",metrics_path="/metrics"}) < 0.15) and (kubelet_volume_stats_inodes_used{namespace=~".*",job="kubelet",metrics_path="/metrics"} > 0)) and (predict_linear(kubelet_volume_stats_inodes_free{namespace=~".*",job="kubelet",metrics_path="/metrics"}[6h], 345600) < 0)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1)) unless on(cluster,namespace,persistentvolumeclaim) (kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1)

Upstream runbook for KubePersistentVolumeInodesFillingUp

KubePersistentVolumeErrors⚓︎

PersistentVolume is having issues with provisioning.

The persistent volume {{ $labels.persistentvolume }} on Cluster {{ $labels.cluster }} has status {{ $labels.phase }}.

Trigger condition

kube_persistentvolume_status_phase{phase=~"Failed|Pending",job="kube-state-metrics"} > 0

Upstream runbook for KubePersistentVolumeErrors

Kubernetes system components⚓︎

Cross-component consistency: version skew between control-plane components, and Kubernetes API clients whose requests are failing.

KubeVersionMismatch⚓︎

Different semantic versions of Kubernetes components running.

There are {{ $value }} different semantic versions of Kubernetes components running on cluster {{ $labels.cluster }}.

Trigger condition

count(count(label_replace(kubernetes_build_info{job!~"kube-dns|coredns"}, "git_version", "$1", "git_version", "(v[0-9]*.[0-9]*).*")) by(git_version,cluster)) by(cluster) > 1

Upstream runbook for KubeVersionMismatch

KubeClientErrors⚓︎

Kubernetes API server client is experiencing errors.

Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors on cluster {{ $labels.cluster }}.

Trigger condition

(sum(rate(rest_client_requests_total{job="apiserver",code=~"5.."}[5m])) by(cluster,instance,job,namespace) / sum(rate(rest_client_requests_total{job="apiserver"}[5m])) by(cluster,instance,job,namespace)) > 0.01

Upstream runbook for KubeClientErrors

Kubernetes API server⚓︎

Availability of the API server itself, certificate expiry for its clients, health of aggregated (extension) APIs, and requests the server is terminating under load. A failure here disables scheduling and rollouts even while running workloads keep serving traffic.

KubeClientCertificateExpiration⚓︎

Client certificate is about to expire.

This alert is defined in 2 variants:

Warning, fires after 5m

A client certificate used to authenticate to kubernetes apiserver is expiring in less than 7.0 days on cluster {{ $labels.cluster }}.

(histogram_quantile(0.01, sum(rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m])) without(namespace,service,endpoint)) < 604800) and on(job,cluster,instance) (apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0)

Critical, fires after 5m

A client certificate used to authenticate to kubernetes apiserver is expiring in less than 24.0 hours on cluster {{ $labels.cluster }}.

(histogram_quantile(0.01, sum(rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m])) without(namespace,service,endpoint)) < 86400) and on(job,cluster,instance) (apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0)

Upstream runbook for KubeClientCertificateExpiration

KubeAggregatedAPIErrors⚓︎

Kubernetes aggregated API has reported errors.

Kubernetes aggregated API {{ $labels.instance }}/{{ $labels.name }} has reported {{ $labels.reason }} errors on cluster {{ $labels.cluster }}.

Trigger condition

sum(increase(aggregator_unavailable_apiservice_total{job="apiserver"}[1m])) by(cluster,instance,name,reason) > 0

Upstream runbook for KubeAggregatedAPIErrors

KubeAggregatedAPIDown⚓︎

Kubernetes aggregated API is down.

Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m on cluster {{ $labels.cluster }}.

Trigger condition

((1 - max(avg_over_time(aggregator_unavailable_apiservice{job="apiserver"}[10m])) by(name,namespace,cluster)) * 100) < 85

Upstream runbook for KubeAggregatedAPIDown

KubeAPIDown⚓︎

Target disappeared from Prometheus target discovery.

KubeAPI has disappeared from Prometheus target discovery.

Trigger condition

absent(up{job="apiserver"})

Upstream runbook for KubeAPIDown

KubeAPIInstanceUnreachable⚓︎

KubeAPI instance is unreachable.

A KubeAPI instance has been unreachable for more than 15 minutes.

Trigger condition

up{job="apiserver"} == 0

Upstream runbook for KubeAPIInstanceUnreachable

KubeAPITerminatedRequests⚓︎

The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.

The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests on cluster {{ $labels.cluster }}.

Trigger condition

(sum(rate(apiserver_request_terminations_total{job="apiserver"}[10m])) by(cluster) / (sum(rate(apiserver_request_total{job="apiserver"}[10m])) by(cluster) + sum(rate(apiserver_request_terminations_total{job="apiserver"}[10m])) by(cluster))) > 0.20

Upstream runbook for KubeAPITerminatedRequests

Kubernetes API server error budget⚓︎

A single alert, defined as four multi-window burn rate variants of the API server's availability SLO. Each variant pairs a long and a short window so that a brief spike does not fire the slow burn rate, and a sustained failure escalates. Read the variants together: the 1h/5m pair means the error budget is being consumed fast enough to exhaust it within roughly two days.

KubeAPIErrorBudgetBurn⚓︎

The API server is burning too much error budget.

This alert is defined in 4 variants:

Critical, fires after 2m

The API server is burning too much error budget on cluster {{ $labels.cluster }}.

(sum(apiserver_request:burnrate1h) by(cluster) > 0.14400000000000002) and on(cluster) (sum(apiserver_request:burnrate5m) by(cluster) > 0.14400000000000002)

Critical, fires after 15m

The API server is burning too much error budget on cluster {{ $labels.cluster }}.

(sum(apiserver_request:burnrate6h) by(cluster) > 0.06) and on(cluster) (sum(apiserver_request:burnrate30m) by(cluster) > 0.06)

Warning, fires after 1h

The API server is burning too much error budget on cluster {{ $labels.cluster }}.

(sum(apiserver_request:burnrate1d) by(cluster) > 0.03) and on(cluster) (sum(apiserver_request:burnrate2h) by(cluster) > 0.03)

Warning, fires after 3h

The API server is burning too much error budget on cluster {{ $labels.cluster }}.

(sum(apiserver_request:burnrate3d) by(cluster) > 0.01) and on(cluster) (sum(apiserver_request:burnrate6h) by(cluster) > 0.01)

Upstream runbook for KubeAPIErrorBudgetBurn

Kubernetes controller manager⚓︎

Reachability of the controller manager. If it is down, controllers stop reconciling — Deployments do not scale, and failed Pods are not replaced.

KubeControllerManagerDown⚓︎

Target disappeared from Prometheus target discovery.

KubeControllerManager has disappeared from Prometheus target discovery.

Trigger condition

absent(up{job="kube-controller-manager"})

Upstream runbook for KubeControllerManagerDown

KubeControllerManagerInstanceUnreachable⚓︎

KubeControllerManager instance is unreachable.

A KubeControllerManager instance has been unreachable for more than 15 minutes.

Trigger condition

up{job="kube-controller-manager"} == 0

Upstream runbook for KubeControllerManagerInstanceUnreachable

Kubernetes scheduler⚓︎

Reachability of the scheduler. If it is down, existing Pods keep running but nothing new is placed on a node.

KubeSchedulerDown⚓︎

Target disappeared from Prometheus target discovery.

KubeScheduler has disappeared from Prometheus target discovery.

Trigger condition

absent(up{job="kube-scheduler"})

Upstream runbook for KubeSchedulerDown

KubeSchedulerInstanceUnreachable⚓︎

KubeScheduler instance is unreachable.

A KubeScheduler instance has been unreachable for more than 15 minutes.

Trigger condition

up{job="kube-scheduler"} == 0

Upstream runbook for KubeSchedulerInstanceUnreachable

Kubelet and node readiness⚓︎

The largest group in the bundle: node conditions, evictions, pod capacity, container runtime and PLEG latency, and kubelet certificate expiry and renewal. This is where a failing node shows up first.

KubeNodeNotReady⚓︎

Node is not ready.

{{ $labels.node }} has been unready for more than 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

(kube_node_status_condition{job="kube-state-metrics",condition="Ready",status="true"} == 0) and on(cluster,node) (kube_node_spec_unschedulable{job="kube-state-metrics"} == 0)

Upstream runbook for KubeNodeNotReady

KubeNodePressure⚓︎

Node has as active Condition.

{{ $labels.node }} on cluster {{ $labels.cluster }} has active Condition {{ $labels.condition }}. This is caused by resource usage exceeding eviction thresholds.

Trigger condition

(kube_node_status_condition{job="kube-state-metrics",condition=~"(MemoryPressure|DiskPressure|PIDPressure)",status="true"} == 1) and on(cluster,node) (kube_node_spec_unschedulable{job="kube-state-metrics"} == 0)

Upstream runbook for KubeNodePressure

KubeNodeUnreachable⚓︎

Node is unreachable.

{{ $labels.node }} is unreachable and some workloads may be rescheduled on cluster {{ $labels.cluster }}.

Trigger condition

(kube_node_spec_taint{job="kube-state-metrics",key="node.kubernetes.io/unreachable",effect="NoSchedule"} unless ignoring(key,value) kube_node_spec_taint{job="kube-state-metrics",key=~"ToBeDeletedByClusterAutoscaler|cloud.google.com/impending-node-termination|aws-node-termination-handler/spot-itn"}) == 1

Upstream runbook for KubeNodeUnreachable

KubeletTooManyPods⚓︎

Kubelet is running at capacity.

Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity on cluster {{ $labels.cluster }}.

Trigger condition

((max(kubelet_running_pods{job="kubelet",metrics_path="/metrics"} > 1) by(cluster,instance) * on(cluster,instance) group_left(node) max(kubelet_node_name{job="kubelet",metrics_path="/metrics"}) by(cluster,instance,node)) / on(cluster,node) group_left() max(kube_node_status_capacity{job="kube-state-metrics",resource="pods"} != 1) by(cluster,node)) > 0.95

Upstream runbook for KubeletTooManyPods

KubeNodeReadinessFlapping⚓︎

Node readiness status is flapping.

The readiness status of node {{ $labels.node }} has changed {{ $value }} times in the last 15 minutes on cluster {{ $labels.cluster }}.

Trigger condition

(sum(changes(kube_node_status_condition{job="kube-state-metrics",status="true",condition="Ready"}[15m])) by(cluster,node) > 2) and on(cluster,node) (kube_node_spec_unschedulable{job="kube-state-metrics"} == 0)

Upstream runbook for KubeNodeReadinessFlapping

KubeNodeEviction⚓︎

Node is evicting pods.

Node {{ $labels.node }} on {{ $labels.cluster }} is evicting Pods due to {{ $labels.eviction_signal }}. Eviction occurs when eviction thresholds are crossed, typically caused by Pods exceeding RAM/ephemeral-storage limits.

Trigger condition

(sum(rate(kubelet_evictions{job="kubelet",metrics_path="/metrics"}[15m])) by(cluster,eviction_signal,instance) * on(cluster,instance) group_left(node) max(kubelet_node_name{job="kubelet",metrics_path="/metrics"}) by(cluster,instance,node)) > 0

Upstream runbook for KubeNodeEviction

KubeletPlegDurationHigh⚓︎

Kubelet Pod Lifecycle Event Generator is taking too long to relist.

The Kubelet Pod Lifecycle Event Generator has a 99th percentile duration of {{ $value }} seconds on node {{ $labels.node }} on cluster {{ $labels.cluster }}.

Trigger condition

node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile{quantile="0.99"} >= 10

Upstream runbook for KubeletPlegDurationHigh

KubeletPodStartUpLatencyHigh⚓︎

Kubelet Pod startup latency is too high.

Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }} on cluster {{ $labels.cluster }}.

Trigger condition

(histogram_quantile(0.99, sum(topk(1, rate(kubelet_pod_worker_duration_seconds_bucket{job="kubelet",metrics_path="/metrics"}[5m])) by(cluster,instance,le,operation_type)) by(cluster,instance,le)) * on(cluster,instance) group_left(node) topk(1, kubelet_node_name{job="kubelet",metrics_path="/metrics"}) by(cluster,instance,node)) > 60

Upstream runbook for KubeletPodStartUpLatencyHigh

KubeletClientCertificateExpiration⚓︎

Kubelet client certificate is about to expire.

This alert is defined in 2 variants:

Warning, fires immediately

Client certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }} on cluster {{ $labels.cluster }}.

kubelet_certificate_manager_client_ttl_seconds < 604800

Critical, fires immediately

Client certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }} on cluster {{ $labels.cluster }}.

kubelet_certificate_manager_client_ttl_seconds < 86400

Upstream runbook for KubeletClientCertificateExpiration

KubeletServerCertificateExpiration⚓︎

Kubelet server certificate is about to expire.

This alert is defined in 2 variants:

Warning, fires immediately

Server certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }} on cluster {{ $labels.cluster }}.

kubelet_certificate_manager_server_ttl_seconds < 604800

Critical, fires immediately

Server certificate for Kubelet on node {{ $labels.node }} expires in {{ $value | humanizeDuration }} on cluster {{ $labels.cluster }}.

kubelet_certificate_manager_server_ttl_seconds < 86400

Upstream runbook for KubeletServerCertificateExpiration

KubeletClientCertificateRenewalErrors⚓︎

Kubelet has failed to renew its client certificate.

Kubelet on node {{ $labels.node }} has failed to renew its client certificate ({{ $value | humanize }} errors in the last 5 minutes) on cluster {{ $labels.cluster }}.

Trigger condition

increase(kubelet_certificate_manager_client_expiration_renew_errors[5m]) > 0

Upstream runbook for KubeletClientCertificateRenewalErrors

KubeletServerCertificateRenewalErrors⚓︎

Kubelet has failed to renew its server certificate.

Kubelet on node {{ $labels.node }} has failed to renew its server certificate ({{ $value | humanize }} errors in the last 5 minutes) on cluster {{ $labels.cluster }}.

Trigger condition

increase(kubelet_server_expiration_renew_errors[5m]) > 0

Upstream runbook for KubeletServerCertificateRenewalErrors

KubeletInstanceUnreachable⚓︎

Kubelet instance is unreachable.

A Kubelet instance has been unreachable for more than 15 minutes.

Trigger condition

up{job="kubelet",metrics_path="/metrics"} == 0

Upstream runbook for KubeletInstanceUnreachable

KubeletDown⚓︎

Target disappeared from Prometheus target discovery.

Kubelet has disappeared from Prometheus target discovery on cluster {{ $labels.cluster }}.

Trigger condition

count(kube_node_info{job="kube-state-metrics"}) by(cluster) unless on(cluster) count(up{job="kubelet",metrics_path="/metrics"} == 1) by(cluster)

Upstream runbook for KubeletDown

kube-state-metrics⚓︎

Health of the exporter that supplies the metrics behind most of the Kubernetes alerts above. Treat these as meta-alerts: while kube-state-metrics is degraded, the workload and resource alerts go quiet whether or not anything is wrong.

KubeStateMetricsListErrors⚓︎

kube-state-metrics is experiencing errors in list operations.

kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.

Trigger condition

(sum(rate(kube_state_metrics_list_total{job="kube-state-metrics",result="error"}[5m])) by(cluster) / sum(rate(kube_state_metrics_list_total{job="kube-state-metrics"}[5m])) by(cluster)) > 0.01

Upstream runbook for KubeStateMetricsListErrors

KubeStateMetricsWatchErrors⚓︎

kube-state-metrics is experiencing errors in watch operations.

kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.

Trigger condition

(sum(rate(kube_state_metrics_watch_total{job="kube-state-metrics",result="error"}[5m])) by(cluster) / sum(rate(kube_state_metrics_watch_total{job="kube-state-metrics"}[5m])) by(cluster)) > 0.01

Upstream runbook for KubeStateMetricsWatchErrors

KubeStateMetricsShardingMismatch⚓︎

kube-state-metrics sharding is misconfigured.

kube-state-metrics pods are running with different --total-shards configuration, some Kubernetes objects may be exposed multiple times or not exposed at all.

Trigger condition

stdvar(kube_state_metrics_total_shards{job="kube-state-metrics"}) by(cluster) != 0

Upstream runbook for KubeStateMetricsShardingMismatch

KubeStateMetricsShardsMissing⚓︎

kube-state-metrics shards are missing.

kube-state-metrics shards are missing, some Kubernetes objects are not being exposed.

Trigger condition

(((2 ^ max(kube_state_metrics_total_shards{job="kube-state-metrics"}) by(cluster)) - 1) - sum(2 ^ max(kube_state_metrics_shard_ordinal{job="kube-state-metrics"}) by(cluster,shard_ordinal)) by(cluster)) != 0

Upstream runbook for KubeStateMetricsShardsMissing

etcd⚓︎

Cluster membership, leader stability, request and fsync latency, and database growth against the storage quota. Losing quorum here takes the whole control plane with it.

etcdMembersDown⚓︎

etcd cluster members are down.

etcd cluster "{{ $labels.job }}": members are down ({{ $value }}).

Trigger condition

max(sum(up{job=~".*etcd.*"} ==bool 0) without(instance) or count(sum(rate(etcd_network_peer_sent_failures_total{job=~".*etcd.*"}[120s])) without(instance) > 0.01) without(To)) without(endpoint) > 0

etcdInsufficientMembers⚓︎

etcd cluster has insufficient number of members.

etcd cluster "{{ $labels.job }}": insufficient members ({{ $value }}).

Trigger condition

sum(up{job=~".*etcd.*"} ==bool 1) without(instance) < ((count(up{job=~".*etcd.*"}) without(instance) + 1) / 2)

etcdNoLeader⚓︎

etcd cluster has no leader.

etcd cluster "{{ $labels.job }}": member {{ $labels.instance }} has no leader.

Trigger condition

etcd_server_has_leader{job=~".*etcd.*"} == 0

etcdHighNumberOfLeaderChanges⚓︎

etcd cluster has high number of leader changes.

etcd cluster "{{ $labels.job }}": {{ $value }} leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.

Trigger condition

increase((max(etcd_server_leader_changes_seen_total{job=~".*etcd.*"}) without(instance) or (0 * absent(etcd_server_leader_changes_seen_total{job=~".*etcd.*"})))[15m:1m]) >= 4

etcdHighNumberOfFailedGRPCRequests⚓︎

etcd cluster has high number of failed grpc requests.

This alert is defined in 2 variants:

—, fires immediately

etcd cluster "{{ $labels.job }}": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.

((100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*",grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[5m])) without(grpc_type,grpc_code)) / sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) without(grpc_type,grpc_code)) > 1

—, fires immediately

etcd cluster "{{ $labels.job }}": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.

((100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*",grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[5m])) without(grpc_type,grpc_code)) / sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) without(grpc_type,grpc_code)) > 5

etcdGRPCRequestsSlow⚓︎

etcd grpc requests are slow

etcd cluster "{{ $labels.job }}": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.

Trigger condition

histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~".*etcd.*",grpc_method!="Defragment",grpc_type="unary"}[5m])) without(grpc_type)) > 0.15

etcdMemberCommunicationSlow⚓︎

etcd cluster member communication is slow.

etcd cluster "{{ $labels.job }}": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.

Trigger condition

histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~".*etcd.*"}[5m])) > 0.15

etcdHighNumberOfFailedProposals⚓︎

etcd cluster has high number of proposal failures.

etcd cluster "{{ $labels.job }}": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.

Trigger condition

rate(etcd_server_proposals_failed_total{job=~".*etcd.*"}[15m]) > 5

etcdHighFsyncDurations⚓︎

etcd cluster 99th percentile fsync durations are too high.

This alert is defined in 2 variants:

—, fires immediately

etcd cluster "{{ $labels.job }}": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.

histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~".*etcd.*"}[5m])) > 0.5

—, fires immediately

etcd cluster "{{ $labels.job }}": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.

histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~".*etcd.*"}[5m])) > 1

etcdHighCommitDurations⚓︎

etcd cluster 99th percentile commit durations are too high.

etcd cluster "{{ $labels.job }}": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.

Trigger condition

histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~".*etcd.*"}[5m])) > 0.25

etcdDatabaseQuotaLowSpace⚓︎

etcd cluster database is running full.

etcd cluster "{{ $labels.job }}": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.

Trigger condition

((last_over_time(etcd_mvcc_db_total_size_in_bytes{job=~".*etcd.*"}[5m]) / last_over_time(etcd_server_quota_backend_bytes{job=~".*etcd.*"}[5m])) * 100) > 95

etcdExcessiveDatabaseGrowth⚓︎

etcd cluster database growing very fast.

etcd cluster "{{ $labels.job }}": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.

Trigger condition

predict_linear(etcd_mvcc_db_total_size_in_bytes{job=~".*etcd.*"}[4h], 14400) > etcd_server_quota_backend_bytes{job=~".*etcd.*"}

etcdDatabaseHighFragmentationRatio⚓︎

etcd database size in use is less than 50% of the actual allocated storage.

etcd cluster "{{ $labels.job }}": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.

Trigger condition

((last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes{job=~".*etcd.*"}[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes{job=~".*etcd.*"}[5m])) < 0.5) and (etcd_mvcc_db_total_size_in_use_in_bytes{job=~".*etcd.*"} > 104857600)

Upstream runbook for etcdDatabaseHighFragmentationRatio

Host and node health⚓︎

Host-level alerts from node_exporter, covering the same ground as the VM alerts on Platform and Infrastructure Alerts but for Kubernetes nodes: filesystem space and inodes, memory, CPU and I/O saturation, clock synchronization, file descriptors, RAID, bonding, and failed systemd units.

NodeFilesystemSpaceFillingUp⚓︎

Filesystem is predicted to run out of space within the next 24 hours.

This alert is defined in 2 variants:

Warning, fires after 1h

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left and is filling up.

((((node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 15) and (predict_linear(node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""}[6h], 86400) < 0)) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Critical, fires after 1h

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left and is filling up fast.

((((node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 10) and (predict_linear(node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""}[6h], 14400) < 0)) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Upstream runbook for NodeFilesystemSpaceFillingUp

NodeFilesystemAlmostOutOfSpace⚓︎

Filesystem has less than 5% space left.

This alert is defined in 2 variants:

Warning, fires after 30m

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left.

(((node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 5) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Critical, fires after 30m

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left.

(((node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 3) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Upstream runbook for NodeFilesystemAlmostOutOfSpace

NodeFilesystemFilesFillingUp⚓︎

Filesystem is predicted to run out of inodes within the next 24 hours.

This alert is defined in 2 variants:

Warning, fires after 1h

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left and is filling up.

((((node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 40) and (predict_linear(node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""}[6h], 86400) < 0)) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Critical, fires after 1h

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left and is filling up fast.

((((node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 20) and (predict_linear(node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""}[6h], 14400) < 0)) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Upstream runbook for NodeFilesystemFilesFillingUp

NodeFilesystemAlmostOutOfFiles⚓︎

Filesystem has less than 5% inodes left.

This alert is defined in 2 variants:

Warning, fires after 1h

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left.

(((node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 5) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Critical, fires after 1h

Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left.

(((node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""}) * 100) < 3) and (node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0)

Upstream runbook for NodeFilesystemAlmostOutOfFiles

NodeNetworkReceiveErrs⚓︎

Network interface is reporting many receive errors.

{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} receive errors in the last two minutes.

Trigger condition

(rate(node_network_receive_errs_total{job="node-exporter"}[2m]) / rate(node_network_receive_packets_total{job="node-exporter"}[2m])) > 0.01

Upstream runbook for NodeNetworkReceiveErrs

NodeNetworkTransmitErrs⚓︎

Network interface is reporting many transmit errors.

{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} transmit errors in the last two minutes.

Trigger condition

(rate(node_network_transmit_errs_total{job="node-exporter"}[2m]) / rate(node_network_transmit_packets_total{job="node-exporter"}[2m])) > 0.01