Skip to content

Monitored Endpoints⚓︎


This page is the reference for everything xView monitors: every scrape job in a standard deployment, what it collects and from which exporter. It follows the metrics along the path described in the xView architecture — from the exporters at a site, through the site collector and the xControl relay, into xView's metric database.

The alerts evaluated over these metrics are documented in Platform and Infrastructure Alerts, xCore Alerts, xControl Alerts and Kubernetes and Monitoring Stack Alerts. The labels every scrape attaches are described in Metric Labels and Conventions, and the exporters themselves in Data Sources and Exporters.

Two tiers of collection⚓︎

Scraping happens in two places, and it is worth keeping them apart when you troubleshoot:

  1. The site collector runs inside each site's cluster, xCore and xRouter alike. It scrapes the local exporters and network functions and keeps a short local history — it is a buffer, not the system of record. Everything it holds is relayed to xControl by the Expeto Agent.
  2. xView's vmagent runs centrally, alongside xControl. It scrapes xControl (both xControl's own metrics and each site's relayed metrics), plus the Kubernetes infrastructure of the cluster it runs in. What it collects is what lands in the metric database and what alerts evaluate against.

A site therefore never needs an inbound connection: the collector is scraped locally, and the agent pushes outward to xControl.

Site collector scrape jobs⚓︎

The site collector discovers most of its targets through the Kubernetes API, keeping only the endpoints that match a named service, so targets follow the cluster's actual inventory without configuration changes. The remaining jobs use the agent's service-discovery endpoint on port 9767, which publishes site-specific targets that Kubernetes cannot enumerate.

Intervals shown as (global) inherit the collector's default rather than setting their own.

Job Monitors Discovery Metrics path Interval
prometheus The site collector itself (self-monitoring) Static: localhost:9090 /metrics 15s
xAgent The Expeto Agent: its own health, plus every network function metric it collects Kubernetes SD (service), keeping .*agent-prom /metrics 15s
node Host health via node_exporter on each cluster node Kubernetes SD (endpoints), keeping .*node-exporter /metrics 15s
k8s Kubernetes object state via kube-state-metrics Kubernetes SD (endpoints), keeping (.+-kube-state-metrics;8080) /metrics 15s
mariadb-galera Site database health via the MariaDB Galera exporter Kubernetes SD (endpoints), keeping (.+-mariadb-galera-metrics;9104) /metrics 15s
hss HSS application metrics from its native Prometheus endpoint Kubernetes SD (endpoints), keeping (.+-hss;8090) /hss/prometheus 15s
udr UDR application metrics from its native Prometheus endpoint Kubernetes SD (endpoints), keeping (.+-udr;.+) /prometheus 15s
udm UDM application metrics from its native Prometheus endpoint Kubernetes SD (endpoints), keeping (.+-udm;.+) /prometheus 15s
ausf AUSF application metrics from its native Prometheus endpoint Kubernetes SD (endpoints), keeping (.+-ausf;.+) /prometheus 15s
other Site-specific targets published by the agent's service-discovery endpoint HTTP SD: http://agent-prom.<namespace>.svc.cluster.local:9767/sd?job=other /metrics 15s
SecGw Security gateway (IPsec concentrator) metrics HTTP SD: http://agent-prom.<namespace>.svc.cluster.local:9767/sd?job=SecGw /metrics 15s
blackbox Synthetic availability probes via the blackbox exporter HTTP SD: http://agent-prom.<namespace>.svc.cluster.local:9767/sd?job=blackbox /metrics 15s (global)

<namespace> in a discovery URL is the namespace the xCore release is installed into.

The other job is the extension point for site-specific monitoring: targets returned by the agent's /sd?job=other endpoint are scraped without changing the managed configuration.

Central xView scrape jobs⚓︎

xView's vmagent defines its scrape jobs as VMScrapeConfig objects rather than a Prometheus configuration file. Three matter:

Job Monitors Discovery Metrics path Interval
site Every site's relayed metrics, one scrape per site HTTP SD against xControl's /sd?job=site endpoint on port 9765 /x-metrics 10s
xControl xControl's own application and JVM metrics Static, xControl port 9765 /metrics 10s
federation A collector scraped directly rather than through the relay Static, collector port 9090 /federate 15s

The site job is the platform's most important scrape. Service discovery returns one entry per site; vmagent rewrites each discovered entry into a site query parameter and directs the request back at xControl, so a single host serves every site's metrics as /x-metrics?site=<id>. Each site's metrics arrive already carrying their own identity labels, which is what lets one alert rule cover every site.

Note

The federation job is not enabled by default. It scrapes a collector's /federate endpoint directly, which is only useful where xView can reach a collector over the network — the relay path through xControl is the norm. See xView architecture.

All three jobs scrape with honored labels, meaning labels already present in the scraped payload are kept in preference to the labels the scrape would otherwise attach.

Kubernetes and monitoring-stack jobs⚓︎

In the cluster where it runs, xView also scrapes the standard Kubernetes control plane and node targets. These are provided by the monitoring stack rather than configured by Expeto:

Target Collected from Notes
kubelet, including cAdvisor Each node's kubelet Scraped every 30s, over HTTPS
Kubernetes API server The kubernetes service
controller-manager, scheduler, etcd Control-plane endpoints Unavailable on managed control planes
CoreDNS The kube-dns service
kube-state-metrics The kube-state-metrics service Kubernetes object state
node_exporter The node-exporter DaemonSet Host-level health
vmagent, vmalert, VMSingle, Alertmanager, Grafana Each component's own metrics endpoint Self-monitoring

Scraping of kube-proxy is disabled by default.

The self-monitoring jobs exist so that a failure of the monitoring stack itself raises an alert — see the vmagent and vmalert alert groups.

Synthetic probes⚓︎

Where scraping a metrics endpoint is not enough — "is this service actually reachable?" — the site collector probes through the blackbox exporter. The blackbox job passes each target to the exporter as a parameter; the exporter performs the check and reports the result as metrics, which the ProbeFailure and ProbeSuccessLow alerts evaluate.

Exporting metrics onward⚓︎

xView does not forward metrics to an external system by default. Where a deployment needs one, a remote-write target is configured on vmagent, and the delivery itself is then monitored by the vmagent alerts — a failing remote write shows up as dropped or retried samples rather than as a silent gap.