Skip to content

Accessing xView⚓︎

xView presents its data through Grafana, which is reachable from a browser at the same hostname as xControl. The rest of the monitoring stack — the metric database, the rule evaluator and Alertmanager — has no public interface by design, and is reached directly when you need it.

Grafana⚓︎

Grafana is served under the /nms path of your xControl hostname:

https://xcontrol.example.com/nms

Signing in⚓︎

Sign in with Expeto Keycloak SSO, using the same identity you use for xControl. Access is granted by role: an account holding the administrator or superuser role gets Grafana's Admin role, which allows creating and editing dashboards. Other accounts get read access to what has been shared with them.

A local administrator account also exists for break-glass access, for cases where the identity provider itself is unavailable. Its credentials are held in a Kubernetes secret named <release>-xview-admin-credentials:

kubectl get secret <release>-xview-admin-credentials -n <namespace> \
  -o jsonpath='{.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d

Warning

The local account bypasses single sign-on and its actions are not attributable to a person. Use SSO for routine work and keep the local credentials for recovery.

Data sources⚓︎

Two data sources are configured and ready to query:

Name Contains
Expeto-NMS The metric database. The default data source — every shipped dashboard uses it.
Expeto-Alertmanager Current alert state, for the alerting views

Shipped dashboards⚓︎

Dashboards are organized into two folders. Expeto's dashboards cover the platform:

Dashboard Shows
Overview The whole deployment at a glance: xControl and xView health, a per-customer summary, and current alerts
Site One site in depth — devices, radios, network function counts, throughput, capacity and host health
Sitev2 A revised per-site view, organized by subsystem: devices, CPU, networking, containers, disk, memory and the agent
xCore Public The xCore itself: resources, quality of service and signalling rates
xControl xControl internals — queues, database, JVM memory, thread pools and error logs
Alerts Firing and pending alerts, with a per-customer breakdown
Infrastructure alerts The same view organized by host rather than by customer
Log Errors Error and fatal log volume, by source, site and logger

The second folder holds the infrastructure dashboards that come with the monitoring stack — Kubernetes cluster, node, pod and namespace views, etcd, CoreDNS, and the monitoring components' own dashboards. These are upstream dashboards, documented by VictoriaMetrics and the Kubernetes mixin projects.

Note

The shipped dashboards are managed by the deployment and are not editable in place — changes would be reverted on the next upgrade. To adapt one, copy it (Dashboard settings → Save as) into a folder of your own and edit the copy.

Running a query⚓︎

Grafana's Explore view is the quickest way to run a PromQL query against the metric database. Select the Expeto-NMS data source and enter an expression, for example:

sum by (site) (xcore_mme_resource_imsi_attached)

Metric Labels and Conventions covers the labels available and the query patterns worth knowing; xCore Metrics lists what there is to query.

Reaching the other components⚓︎

The metric database, the rule evaluator, Alertmanager and the scraper are internal services with no ingress. Reach them with a port-forward when you need to inspect them directly — most often to confirm that a scrape target is healthy, or that a rule is loaded and evaluating.

Forward the metric database:

kubectl port-forward -n <namespace> svc/vmsingle-<release>-victoria-metrics-k8s-stack 8428:8428

Then browse to http://localhost:8428/vmui for an interactive query interface, or query the API directly:

curl -s 'http://localhost:8428/api/v1/query?query=up{job="site"}' | python3 -m json.tool

The other components follow the same pattern:

Component Service Port Useful for
Metric database vmsingle-<release>-victoria-metrics-k8s-stack 8428 Queries, the vmui interface, checking what data exists
Scraper vmagent-<release>-victoria-metrics-k8s-stack 8429 Scrape target health — which targets are up, and why one is failing
Rule evaluator vmalert-<release>-victoria-metrics-k8s-stack 8080 Which rules are loaded, and which are firing
Alertmanager vmalertmanager-<release>-victoria-metrics-k8s-stack 9093 Current alerts, silences and routing

Tip

When a site's metrics are missing, the scraper's targets page is the first place to look. It reports the last scrape result per target, which distinguishes "xView never tried" from "xView tried and the request failed" — and those have entirely different causes.

Checking which sites are reporting⚓︎

A quick health check across the estate, run against the metric database:

up{job="site"}

Every site should return 1. A site returning 0 is not relaying metrics — see the failure path in xView Architecture — and will be raising xCoreAgentOutOfContact.

To see how fresh each site's data is:

time() - max by (site) (timestamp(up{job="site"} == 1))

Alert delivery⚓︎

Firing alerts are sent to Alertmanager, which routes them onward. Routing is configured per deployment. Until a receiver is configured, alerts fire and are visible in Grafana and in the rule evaluator, but no notification is sent.

Configuring this is Alertmanager's own domain — its configuration reference covers receivers, routing trees, grouping and inhibition. Route on the severity label, and use the customer and site labels to direct alerts to whoever is responsible for that part of the estate. Contact Expeto support to have this wired into your notification or ticketing system.

To add alerts of your own, see Custom Alerts and Rules.