Restore with point-in-time recovery¶
Point-in-time recovery (PITR) replays binary logs (binlogs) on top of a full backup so you can bring your database to an exact moment or an exact transaction, instead of only the moment the backup finished.
You can make a point-in-time restore:
- in-place, on the same cluster
- to a side cluster
- to a new cluster
- after a disaster
Set spec.pxcCluster to whichever target you’re restoring into, then follow the steps on this page for the pitr section — everything you need for a PITR restore, for any of these destinations, is on this one page.
Before you start, enable point-in-time recovery on the source cluster so binlogs are already uploaded to storage.
Choose a recovery target¶
Set pitr.type to one of four values, depending on where you want recovery to stop and what you already know about the incident:
pitr.type |
Use it when… | What you need |
|---|---|---|
date |
You know roughly when the incident happened (for example, from an application alert or a user report) and an exact transaction boundary doesn’t matter. | date key, format "YYYY-MM-DD HH:MM:SS", not later than the backup’s status.latestRestorableTime — see Before you restore for how to check it |
transaction |
You know exactly which transaction caused the problem and want to stop right before it, or you need the restore to land on an exact, reproducible boundary rather than an approximate time. | gtid key: the exact GTID of the transaction following the last one to keep — see Find a GTID for point-in-time recovery if you don’t already know it |
skip |
You want to remove a single bad statement — a bad migration, an errant UPDATE — while keeping everything that came after it. |
gtid key: the exact GTID or GTID set to exclude — see Find a GTID for point-in-time recovery if you don’t already know it |
latest |
You’ve had an infrastructure loss and want to recover as close to the present moment as possible, with no specific transaction known to be bad. | Nothing beyond the backup and binlog storage |
If you chose transaction or skip, see Find a GTID for point-in-time recovery to look up the GTID before you continue. If you chose date or latest, skip ahead to Before you restore.
Before you restore¶
-
Export the namespace as an environment variable. Replace the
<namespace>placeholder with your value:export NAMESPACE=<namespace> -
Check how far forward you can restore. The Operator tracks the latest point it can guarantee a consistent recovery as
status.latestRestorableTimeon the Backup object:kubectl get pxc-backup <backup_name> -n <namespace> -o jsonpath='{.status.latestRestorableTime}'For a
date-type restore, yourdatevalue can’t be later than this. The value only appears once the backup has succeeded and the Operator has confirmed there’s no binlog gap blocking recovery up to that point — see Binlog gaps if it’s missing or you suspect it’s stale. If you’re restoring bytransactionorskipinstead, see Find a GTID for point-in-time recovery to look up the GTID you need. -
Disable point-in-time recovery on the target cluster before restoring, regardless of whether the backup itself was made with PITR enabled or not:
kubectl patch pxc cluster1 \ -n <namespace> \ --type merge \ -p '{"spec":{"backup":{"pitr":{"enabled":false}}}}' -
The Operator still requires a Secret with the same user passwords used at backup time for a PITR restore — this is a known limitation, unlike a full-backup restore, which tolerates changed passwords since Operator 1.18.0. If passwords have since changed, follow Restore the cluster when a backup has different passwords before you restore.
Run the restore¶
Point-in-time recovery requires the full backup itself to be in cloud storage — see Considerations. A Persistent Volume backup can’t be the base for a PITR restore.
Pick the section that matches how you’re addressing the full backup. See Choose backupName or backupSource for what these mean and when each applies — the keys below only add the pitr section on top of that choice.
Use backupName¶
Use this when restoring in-place or to a side cluster in the same namespace as the source — a Backup object for the full backup already exists there.
-
Set the following keys for the
PerconaXtraDBClusterRestoreCustom Resource:spec.pxcCluster: the name of the target clusterspec.backupName: the name of your backupspec.pitr: the recovery target you chose above, plus where to find the binlogs — eitherpitr.backupSource.storageName, referencing a storage you’ve already defined, or the storage settings written out directly underpitr.backupSource
-
Pass this configuration to the Operator:
apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore1 spec: pxcCluster: cluster1 backupName: backup1 pitr: type: date date: "2020-12-31 09:37:13" backupSource: storageName: s3-us-westkubectl apply -f deploy/backup/restore.yaml -n <namespace>cat <<EOF | kubectl apply -f- apiVersion: "pxc.percona.com/v1" kind: "PerconaXtraDBClusterRestore" metadata: name: "restore1" spec: pxcCluster: "cluster1" backupName: "backup1" pitr: type: date date: "2020-12-31 09:37:13" backupSource: storageName: "s3-us-west" EOF
Use backupSource, storage already defined on the target¶
Use this when restoring to a side cluster in a different namespace, or to a new cluster, and the backup storage is already configured in the target cluster’s cr.yaml.
-
Set the following keys for the
PerconaXtraDBClusterRestoreCustom Resource:spec.pxcCluster: the name of the target clusterspec.backupSource.storageName: the name matching thebackup.storagessubsection of the target cluster’sdeploy/cr.yamlspec.backupSource.destination: the backup’s location, fromkubectl get pxc-backupon the source clusterspec.pitr: the recovery target you chose above, pluspitr.backupSource.storageNamefor the binlog storage
-
Pass this configuration to the Operator:
apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore1 spec: pxcCluster: cluster2 backupSource: storageName: s3-us-west destination: s3://S3-BUCKET-NAME/BACKUP-NAME pitr: type: date date: "2020-12-31 09:37:13" backupSource: storageName: s3-us-westapiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore1 spec: pxcCluster: cluster1 backupSource: storageName: azure destination: azure://AZURE-CONTAINER-NAME/BACKUP-NAME pitr: type: date date: "2020-12-31 09:37:13" backupSource: storageName: azurekubectl apply -f deploy/backup/restore.yaml -n <namespace>
Use backupSource, full inline storage details¶
Use this when the target cluster has no storage configuration to reference — restoring after a disaster, or to a new cluster where you haven’t predefined the storage.
-
Set the following keys for the
PerconaXtraDBClusterRestoreCustom Resource:spec.pxcCluster: the name of the target clusterspec.backupSource: the full backup’s location and storage settings — adestinationkey (fromkubectl get pxc-backup -n <namespace>on the source cluster) plus the necessary storage configuration keys, just like in the source cluster’sdeploy/cr.yamlspec.pitr: the recovery target you chose above, pluspitr.backupSourcewith the binlog storage settings — these must exactly match the storage used on the source cluster (credentials, endpoint, bucket name, and so on)
-
Pass this configuration to the Operator:
apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore1 spec: pxcCluster: cluster1 backupSource: destination: s3://S3-BUCKET-NAME/BACKUP-NAME s3: bucket: S3-BUCKET-NAME credentialsSecret: my-cluster-name-backup-s3 region: us-west-2 pitr: type: date date: "2020-12-31 09:37:13" backupSource: s3: bucket: S3-BINLOG-BACKUP-BUCKET-NAME-HERE credentialsSecret: my-cluster-name-backup-s3 region: us-west-2apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore1 spec: pxcCluster: cluster1 backupSource: destination: azure://AZURE-CONTAINER-NAME/BACKUP-NAME azure: container: AZURE-CONTAINER-NAME credentialsSecret: my-cluster-azure-secret pitr: type: date date: "2020-12-31 09:37:13" backupSource: azure: container: AZURE-BINLOG-CONTAINER-NAME-HERE credentialsSecret: my-cluster-azure-secret -
Apply the configuration to start the restore:
kubectl apply -f deploy/backup/restore.yaml -n <namespace>
Verify the restore¶
-
Confirm the Restore object reports
Succeeded:kubectl get pxc-restore -n $NAMESPACE -
Confirm the cluster reports the
readystatus:kubectl get pxc -n $NAMESPACE -
Confirm the data landed where you expected. Check the last applied transaction against the GTID or date you targeted:
mysql> SHOW GLOBAL VARIABLES LIKE 'gtid_executed';For a
date-type restore, also spot-check a row you know should (or shouldn’t) be present based on your target time.
Post-restore steps¶
-
Configure the main storage within the target cluster’s
cr.yaml, then re-enable point-in-time recovery to start binlog collection again. -
If you restored in-place, remove the recovery record. The restore creates a new Galera cluster UUID, and without this step the next full cluster crash can be refused.
-
Make a new full backup once you’ve confirmed the restore is correct. The restored database is now the new baseline for future recoveries — don’t rely on replaying binlogs past this point from the old backup chain.
Binlog gaps¶
The Operator monitors the binlog gaps detected by the binlog collector, if any. If a backup contains such gaps, the Operator marks the status of the latest successful backup with a condition that indicates the backup can’t guarantee consistent point-in-time recovery:
To watch for gaps before they affect a restore, monitor the pxc_binlog_collector_gap_detected_total metric — see Binary logs statistics.
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterBackup
metadata:
name: backup1
spec:
pxcCluster: pitr
storageName: minio
status:
completed: "2022-11-25T15:57:29Z"
conditions:
- lastTransitionTime: "2022-11-25T15:57:48Z"
message: Binlog with GTID set e41eb219-6cd8-11ed-94c8-9ebf697d3d20:21-22 not found
reason: BinlogGapDetected
status: "False"
type: PITRReady
state: Succeeded
Trying a point-in-time restore from such a backup results in the following error:
Backup doesn't guarantee consistent recovery with PITR. Annotate PerconaXtraDBClusterRestore with percona.com/unsafe-pitr to force it.
You can bypass this check and force the restore by annotating the Restore object with percona.com/unsafe-pitr:
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterRestore
metadata:
annotations:
percona.com/unsafe-pitr: "true"
name: restore2
spec:
pxcCluster: pitr
backupName: backup1
pitr:
type: latest
backupSource:
storageName: "minio-binlogs"
Created: October 21, 2026