How to back up the OPCP controller data

Objective

OPCP backs up its controller data, so you can recover from physical or logical failures. These backups do not cover any other data, such as the data stored on the OPCP servers.

Backups of OPCP controller data are automatically scheduled hourly for databases, PVs, KMS autounsealer shares and tfstate. Database backups rely on the Kubernetes ScheduledBackups resource; PV, KMS autounsealer share and tfstate backups rely on Velero.

The backups of tfstate contain the state of OpenTofu within the setup of OPCP. OpenTofu uses it, for example, to set up IAM.

PVs are the Kubernetes Persistent Volume resources. This is stateful data stored in the controller's Kubernetes cluster.

The KMS autounsealer shares are secrets needed to unseal the secrets management system.

Requirements

Make sure your backups are stored outside the cluster: run opcp-cli config edit and check that backups.endpoint.url points outside the cluster. Also backups.endpoint.region must be set accordingly. The buckets configured in backups.db.bucket and backups.pv.bucket must exist.

Credentials must also be set correctly with opcp-cli secrets passwords --edit. The keys stringData.backup_access_key_id and stringData.backup_secret_access_key must be set with the secrets provided by the S3¹ provider. Make sure the user has access to all the buckets.

Instructions

You can trigger backups manually from the terminal.

Quick procedure

Run the following lines as root (using sudo -i) on the OPCP controller node.

kubectl get cluster -A -o jsonpath='{range .items[*]}kubectl cnpg backup {.metadata.name} -n {.metadata.namespace}{"\n"}{end}' | bash # schedule backups NOW for all database clusters
velero get schedule -o json | jq -r '.items[] | .metadata.name as $name | {cmd: ("velero backup create --from-schedule " + $name)} | .cmd' | bash # schedule backups NOW for all resources backed up by velero

Verify by running the following on the OPCP controller node.

kubectl get backup -A
velero backup get

Store your KMS shares:

touch kms-shares-backup.yaml
for i in {1..3}; do
  echo "---" >> kms-shares-backup.yaml
  kubectl -n kms get secret ovhcloud-kms-share-${i} -o yaml >> kms-shares-backup.yaml
done

Store the contents of kms-shares-backup.yaml in a safe place, ideally in your trusted password manager.

Databases

To list all databases, run kubectl get cluster -A. Do a backup by running kubectl cnpg backup {db-name} -n {namespace}. Replace {db-name} with the database cluster name and {namespace} with the corresponding namespace.

There is a shortcut to trigger backups of all the database clusters: kubectl get cluster -A -o jsonpath='{range .items[*]}kubectl cnpg backup {.metadata.name} -n {.metadata.namespace}{"\n"}{end}' | bash.

KMS autounsealer shares

Create a backup of kms-autounsealer-shares-backup:

velero backup create --from-schedule kms-autounsealer-shares-backup

PVs

Create a backup of the PVs:

velero backup create --from-schedule pvs-backup

tfstate

OPCP relies on OpenTofu during setup. Its state must be backed up. Create a backup by running:

velero backup create --from-schedule tfstate-backup

Verification

To check the database backups, run kubectl get backup -A: it lists all available database backups.

To verify PV, KMS autounsealer shares and tfstate backups are present, run velero backup get.

Also check your S3 buckets, for example with s3cmd: if they contain no data, the backup configuration is wrong. First configure s3cmd to read from your bucket:

cat << EOF > ~/.s3cfg
[default]
access_key = ${ACCESS_KEY_ID}
secret_key = ${SECRET_ACCESS_KEY}
bucket_location = ${REGION}
host_base = s3.gra.io.cloud.ovh.net
host_bucket = %(bucket)s.s3.gra.io.cloud.ovh.net
EOF

Adjust the values as needed. Then run s3cmd ls s3://bucket-name to check that the bucket contains data.

KMS unsealer shares

The KMS deployed in OPCP is based on Shamir's secret sharing: the secret is divided into three shares, two of which are required for decryption. As of OPCP version 3.1.0, these shares are not backed up, as they are considered highly sensitive.

To back them up, store the Kubernetes secrets within your trusted password manager, completely separate from the OPCP installation.

You can retrieve the secrets by running:

kubectl -n kms get secret -o yaml ovhcloud-kms-share-1
kubectl -n kms get secret -o yaml ovhcloud-kms-share-2
kubectl -n kms get secret -o yaml ovhcloud-kms-share-3

Store the output in a safe place, ideally in your password manager.

Go further

For more information about how to configure your backups, see the How to update the backup S3 buckets guide.

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

Join our community of users.

¹: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.