Skip to content

Custom Alerts and Rules⚓︎

xView ships a default set of alerts covering the platform and the infrastructure it runs on. You can add your own alongside them, without modifying anything Expeto ships and without a redeploy.

Alerts are Kubernetes resources. You write a rule as a VMRule object, apply it with kubectl, and xView's rule evaluator picks it up automatically — it selects every VMRule in the cluster, in any namespace. The same mechanism defines recording rules, which precompute an expression on a schedule.

Before writing a rule, read Metric Labels and Conventions. Rules that use the platform's identity labels work across every site and customer; rules that hard-code a site name have to be rewritten each time the estate changes.

Writing a rule⚓︎

A VMRule holds one or more named groups, each containing rules. The rule body is ordinary Prometheus alerting-rule syntax, so any rule you have written for Prometheus transfers unchanged.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
  name: acmecorp-custom-alerts
  namespace: expeto-monitoring
spec:
  groups:
    - name: AcmeCorpCapacity
      rules:
        - alert: SiteAttachFailureRateHigh
          expr: |
            sum by (site, siteId, customer, customerId) (
              rate(xcore_mme_nas_total{message="att", type="reject",
                                       site_state!~"decommissioned|maintenance"}[10m])
            )
            /
            sum by (site, siteId, customer, customerId) (
              rate(xcore_mme_nas_total{message="att", type="request",
                                       site_state!~"decommissioned|maintenance"}[10m])
            ) > 0.05
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Attach failure rate above 5% at {{ $labels.site }} for {{ $labels.customer }}"
            description: >-
              More than 5 percent of attach requests at {{ $labels.site }}({{ $labels.siteId }})
              for {{ $labels.customer }}({{ $labels.customerId }}) have been rejected for the
              last 15 minutes.

Applied with:

kubectl apply -f acmecorp-custom-alerts.yaml

Fields⚓︎

Field Required Purpose
spec.groups[].name yes Names the group. Rules in a group are evaluated in order, so a recording rule can feed a later rule in the same group.
spec.groups[].interval no How often to evaluate this group. Defaults to the deployment's evaluation interval.
rules[].alert one of The alert name. Use PascalCase, matching the shipped alerts.
rules[].record one of The name of a recording rule's output series instead of an alert.
rules[].expr yes The PromQL expression. An alert fires for every series the expression returns.
rules[].for no How long the condition must hold before firing. Omit for immediate.
rules[].labels no Labels added to the alert. Set severity here.
rules[].annotations no Human-readable text. summary and description are the conventional keys.

Conventions worth following⚓︎

The shipped rules follow a handful of conventions. Matching them means your alerts group, route and read the same way as everything else:

  • Set severity to critical, warning or info. Alert routing is usually configured on this label, so an alert with a severity outside that set may not be delivered at all.
  • Exclude suspended sites with site_state!~"decommissioned|maintenance" in the expression. Without it your alert will fire for sites that are deliberately down for maintenance.
  • Carry identity through the aggregation. Group by site, siteId, customer and customerId so the labels are available to the annotations — an alert that has been aggregated down to a single number cannot say which site it is about.
  • Write annotations that name the site and customer, using {{ $labels.site }} and {{ $labels.customer }}, with the identifiers in the description. This is what makes an alert actionable in a notification that carries no other context.
  • Put the reasoning in description, and keep summary to one line.

Warning

Do not edit the alert rules xView ships. They are managed by the Helm release and any change will be reverted on the next upgrade. To change one, copy it into a VMRule of your own and adjust it there — and be aware both will then evaluate, so also arrange to suppress the original if you do not want it firing alongside yours.

Checking that a rule was accepted⚓︎

The operator validates each VMRule as it is applied and reports the outcome on the object itself. This is the fastest way to catch a malformed expression:

kubectl get vmrules -A

The output carries Status and Sync Error columns. A rule that failed to parse shows the error there:

kubectl describe vmrule acmecorp-custom-alerts

An accepted rule is loaded within a few seconds. To confirm it is being evaluated, and to see whether it is currently firing, open the rule evaluator's interface or the Alerting section of Grafana — see Accessing xView.

Tip

Develop the expression before you wrap it in a rule. Run it as a query first, in Grafana's Explore view or against the metric database directly, and confirm it returns the series you expect — including the identity labels your annotations reference. A rule whose expression returns nothing never fires and looks identical to one that is working.

Recording rules⚓︎

A recording rule evaluates an expression on a schedule and stores the result as a new series. Use one when an expression is expensive, or when several dashboards and alerts share the same subexpression:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
  name: acmecorp-recording-rules
  namespace: expeto-monitoring
spec:
  groups:
    - name: AcmeCorpAggregates
      interval: 1m
      rules:
        - record: site:attach_requests:rate5m
          expr: |
            sum by (site, siteId, customer, customerId) (
              rate(xcore_mme_nas_total{message="att", type="request"}[5m])
            )

The convention for a recorded series name is level:metric:operation — here, an attach-request rate aggregated to the site level. The result is queryable like any other metric.

Alerts written as PrometheusRule⚓︎

xView also accepts standard PrometheusRule objects. They are converted automatically, so existing rules from a Prometheus-based system can be applied unchanged:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: acmecorp-imported-rules
  namespace: expeto-monitoring
spec:
  groups:
    - name: AcmeCorpImported
      rules:
        - alert: ExampleAlert
          expr: up == 0
          for: 5m
          labels:
            severity: warning

For new work prefer VMRule — it is the native form, and the converted copy of a PrometheusRule is what actually gets evaluated, which is one more indirection to reason about when something is wrong.

Making sure an alert reaches someone⚓︎

A firing alert is sent to Alertmanager, which decides what to do with it. Alert delivery is configured per deployment: until Alertmanager has a receiver configured for your environment, an alert will fire and be visible in the interfaces without notifying anyone.

Configuring receivers, routing and grouping is Alertmanager's own domain, and its configuration documentation is the reference. Route on the severity label, and use the customer and site labels to direct alerts to the team responsible for that part of the estate. Talk to Expeto support about wiring this into your existing notification or ticketing system.