Skip to content

Upgrade, Backup and Geo-Redundancy⚓︎

This page covers the day-two operation of an xControl installation: upgrading the release, backing up and restoring the database, running a hot standby at a second site, and switching roles between sites. Everything here is driven by the same values.yaml you installed from.

Upgrade⚓︎

Upgrade by re-running Helm with the same values file and the credentials:

helm repo update
helm upgrade xcontrol expeto/xcontrol -n xcontrol -f values.yaml \
    --set imageCredentials.username='<username>' \
    --set imageCredentials.password='<password>' \
    --set imageCredentials.email='user@example.com'

The values file is yours: it holds every setting of the installation, lives in version control, and is passed on every upgrade together with the credentials, exactly as at install time. Anything it does not mention takes the chart's default for the new release. If no values file is kept, helm upgrade xcontrol expeto/xcontrol -n xcontrol --reuse-values carries the current release's values forward, credentials included.

Database migrations run automatically before the new API server starts. Pin a version with --version when you need to control exactly which chart is installed, and review helm diff output or a helm template render before upgrading a production installation.

Important

Never change a running installation with kubectl edit or kubectl scale. The next helm upgrade re-applies the values file and reverts the change. Put every setting in values.yaml.

Adopt the CustomResourceDefinitions once⚓︎

From release 3.3.22 the chart installs and upgrades the CustomResourceDefinitions of the operators it bundles itself: CloudNativePG, the Prometheus Operator and, with xView, VictoriaMetrics. Helm installs them on the first install but does not own them yet, so the first upgrade after an installation of 3.3.22 or later, and the upgrade from 3.3.21 or earlier to 3.3.22, stops before changing anything and prints the command that adopts them. Run that one upgrade with --take-ownership and global.crds.ignoreOwnershipCheck=true, which needs Helm 3.17 or later:

helm upgrade xcontrol expeto/xcontrol -n xcontrol -f values.yaml --take-ownership \
    --set global.crds.ignoreOwnershipCheck=true \
    --set imageCredentials.username='<username>' \
    --set imageCredentials.password='<password>' \
    --set imageCredentials.email='user@example.com'

Later upgrades need neither setting. A CustomResourceDefinition that another Helm release or OLM manages stops the upgrade with a message naming the manager; on OpenShift the platform keeps the Prometheus Operator's and the chart leaves them alone.

Scaling the API server⚓︎

The API server is the only stateless tier you scale. Enable the Horizontal Pod Autoscaler in values.yaml:

deployments:
  api:
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 4
      targetCPUUtilizationPercentage: 80

Keycloak, ActiveMQ and the database are sized through their own sections of the values file; see the configuration reference and the sizing guide.

Database backups⚓︎

The PostgreSQL cluster is managed by the CloudNativePG operator, which archives write-ahead logs continuously and takes scheduled base backups to S3-compatible object storage when backups are enabled:

cluster:
  backups:
    enabled: true
    provider: s3
    endpointURL: https://s3.example.com
    s3:
      region: us-east-1
      bucket: xcontrol-backups
      path: /production
      accessKey: <access key>
      secretKey: <secret key>
    scheduledBackups:
      - name: daily
        schedule: "0 0 0 * * *"      # six fields, seconds first: daily at midnight
        backupOwnerReference: self
    retentionPolicy: 30d

Check the state of backups with the operator's resources:

kubectl -n xcontrol get backups.postgresql.cnpg.io
kubectl -n xcontrol get scheduledbackups.postgresql.cnpg.io

Keep the values.yaml and the Secrets it generated (kubectl get secret -n xcontrol -o yaml) alongside the backups. Together they are sufficient to rebuild the installation.

Restore⚓︎

To rebuild an installation from object storage, install a new release with cluster.mode set to recovery and point it at the backup:

cluster:
  mode: recovery
  recovery:
    method: object_store
    clusterName: xcontrol-cluster          # name of the cluster that wrote the backups
    provider: s3
    endpointURL: https://s3.example.com
    s3:
      region: us-east-1
      bucket: xcontrol-backups
      path: /production
      accessKey: <access key>
      secretKey: <secret key>

The operator restores the latest base backup, replays the archived logs and then hands the cluster to the application. Because the application passwords are stored in the database, set global.database.passwords (or encodedPasswords) to the values of the original installation so the new Secrets match the restored roles.

Hot standby at a second site⚓︎

An xControl installation can run a streaming replica of its database in another Kubernetes cluster. The replica site runs the full chart, with its application pods held back until its database is promoted, so it becomes a working xControl within minutes of a failover. Local failover inside one cluster (a lost node or instance) needs none of this: the operator promotes a synchronous standby automatically without data loss.

The primary's Helm release notes (helm get notes xcontrol -n xcontrol) print the exact command for its replica, including the encoded passwords the replica must share. In outline:

  1. Copy the shared certificate authority Secret expeto-shared-ca from the primary's namespace to the replica's cluster and namespace.
  2. Install the chart at the replica site with cluster.mode=replica, cluster.replica.primary set to the primary's release name, the pg_basebackup origin pointing at the primary's externally reachable PostgreSQL host, and cluster.cluster.certificates.serverAltDNSNames set to the replica's own externally reachable host.
  3. Re-run that command after every upgrade of the primary so the replica's credentials stay in step.

The kubectl-xcontrol plugin performs the same steps in one command:

kubectl xcontrol init                                  # record both sites' contexts and namespaces
kubectl xcontrol install-replica --to <standby site> --chart expeto/xcontrol -- -f values.yaml
kubectl xcontrol status                                # roles and health of both sites

Switchover and failover⚓︎

Role changes between sites are always run with the plugin; it patches the CloudNativePG cluster resources directly and guarantees the ordering.

Situation Action
Planned maintenance at the primary site kubectl xcontrol switchover --to <site> demotes the primary cleanly and promotes the replica with zero data loss.
Primary site lost kubectl xcontrol failover --to <site> --confirm promotes the replica. Writes not yet replicated are lost.
Recovered former primary kubectl xcontrol rejoin --site <site> --confirm re-bootstraps it as a replica. Never let a fenced former primary take writes again without this step.
Database pod or node lost inside one site Nothing. The operator promotes a local standby automatically.
Sites partitioned but the replica still streams Nothing. Do not force a failover during a partition.

After a switchover or failover the cluster resources no longer match each release's declared cluster.mode. Re-run the helm upgrade for both releases with the roles swapped in their values at the next opportunity; an upgrade with the old values would otherwise re-assert the old roles.

Failover witness⚓︎

The chart can deploy a witness (xcontrol-operator.enabled=true) at a third location that watches both sites. In monitor mode it counts down a failing primary and alerts; a human then runs the failover after checking the blast radius. witness mode arms automatic failover after config.witness.failAfter. Arm it only from a genuine third location, after failover drills and with an alert path for every witness log line: with two sites there is no quorum, a witness that cannot reach the primary cannot tell a dead site from a network partition, and a promotion during a partition creates a split brain. The witness never fails back; returning a site to service is always a deliberate step.

helm install failover-witness expeto/xcontrol -n xcontrol-witness --create-namespace \
    --set xcontrol-operator.enabled=true \
    --set xcontrol-operator.mode=monitor \
    --set xcontrol-operator.config.sites.site-a.namespace=xcontrol \
    --set xcontrol-operator.config.sites.site-a.cluster=xcontrol-cluster \
    --set xcontrol-operator.config.sites.site-a.host=xcontrol-cluster-rw.xcontrol.svc.cluster.local \
    --set xcontrol-operator.config.sites.site-b.namespace=xcontrol \
    --set xcontrol-operator.config.sites.site-b.cluster=xcontrol-standby-cluster \
    --set xcontrol-operator.config.sites.site-b.host=xcontrol-standby-cluster-rw.xcontrol.svc.cluster.local \
    --set xcontrol-operator.config.sites.site-b.kubeconfigSecret=site-b-kubeconfig \
    --set xcontrol-operator.config.witness.failAfter=3m

Sites in the witness's own cluster are reached with its service account; remote clusters through kubeconfigs mounted from Secrets.

Uninstall⚓︎

helm uninstall xcontrol -n xcontrol
kubectl delete namespace xcontrol

Helm does not remove the CloudNativePG CustomResourceDefinitions or the PersistentVolumeClaims of the database. Delete the claims only when you are certain the data is no longer needed, and keep the CustomResourceDefinitions if another release on the cluster uses the operator.