Swap roles between the Main and DR sites¶
A planned switchover turns the DR site into the primary and the Main site into the standby, while both sites stay healthy. Use it for maintenance, for a disaster recovery drill, or to move the primary to another region. If the Main site is already lost, promote the standby instead.
Stop application writes to the Main site before you start. The swap then loses no data. Writes stay unavailable for about 15 to 25 seconds, plus the time the DR site needs to replay any WAL that it hasn’t applied yet.
What happens to the old primary¶
The type of your standby cluster decides what the Main site does after DR is promoted:
| Standby type | Main site after the swap | What you do |
|---|---|---|
| Streaming, or streaming with a repository | Follows the new primary on its own | Nothing. Check with patronictl list |
| Repo-based | Stays on its old timeline and can’t follow DR | Rebuild the Main site |
A repository-based Main site cannot follow because the shutdown checkpoint never reaches the pgBackRest repository. This checkpoint is the final record that PostgreSQL writes when it stops. The DR site never replays that record, so the two sites end up on different timelines. If you plan regular swaps, use a streaming standby and avoid the rebuild.
Before you start¶
The examples use the main cluster and the dr cluster from the streaming replication tutorial. Set the namespaces of your clusters once, and you can run the commands as they are:
export MAIN_NAMESPACE=main-pg
export DR_NAMESPACE=dr-pg
-
Check that the DR cluster is
readyand that theStandbyLaggingcondition isn’tTrue:kubectl get pg dr -n "${DR_NAMESPACE}" -
For a streaming standby, check that the Main site can reach the DR
-haService and trusts its certificates. The certificates of both sites must be signed by the same CA. -
Make sure that the DR cluster has the same user passwords and the same pgBouncer password as the Main site. When the Operator promotes DR, it sets the password of each user in
spec.usersand of the pgBouncer user to the value in the DR Secret. Follow Prepare the user passwords for each user, and Copy the pgBouncer password. When the passwords are the same on both sites, pgBouncer can authenticate in the new primary. If the passwords differ, see Swap back. -
Make sure you can switch application traffic to the DR site after the swap.
Swap the roles¶
-
Stop application writes to the Main site.
The Operator doesn’t stop two clusters from running as primary at the same time. Writes that both sites accept during an overlap are lost on the site that you demote.
-
Get the Main primary Pod and record its current WAL position:
MAIN_PRIMARY=$(kubectl get pod -n "${MAIN_NAMESPACE}" \ -l postgres-operator.crunchydata.com/cluster=main,postgres-operator.crunchydata.com/role=primary \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n "${MAIN_NAMESPACE}" "${MAIN_PRIMARY}" -c database -- \ psql -t -A -c "SELECT pg_current_wal_lsn();"Sample output
0/7034CF8Save this value. Read it after you stop the writes and before you demote the Main site. A value that you read after the demotion is never reached by a repo-based standby, and the wait in step 5 never ends.
You don’t need to run
pg_switch_wal(). When the Main site shuts down cleanly during the demotion, PostgreSQL switches to a new WAL segment and archives it, so the DR site receives all the writes. -
Demote the Main site. Set
standby.enabledtotrueand point the cluster to the repository. For a streaming standby, also setstandby.hostto the DR-haService:kubectl patch pg main -n "${MAIN_NAMESPACE}" --type=merge \ -p '{"spec":{"standby":{"enabled":true,"repoName":"repo1","host":null,"port":null}}}'The
nullvalues remove ahostthat the cluster kept from an earlier standby setup, so that the Main site reads only from the repository.kubectl patch pg main -n "${MAIN_NAMESPACE}" --type=merge \ -p "{\"spec\":{\"standby\":{\"enabled\":true,\"repoName\":\"repo1\",\"host\":\"dr-ha.${DR_NAMESPACE}.svc\",\"port\":5432}}}"If your standby has no repository, omit
repoName. -
Wait for the Main site to stop accepting writes:
kubectl exec -n "${MAIN_NAMESPACE}" "${MAIN_PRIMARY}" -c database -- \ psql -t -A -c "SELECT pg_is_in_recovery();"Don’t continue until the command returns
t. -
Wait until the DR site replays the position that you recorded in step 2. Get the DR primary Pod, then run the check until it returns
t. Replace<lsn>with the saved value:DR_PRIMARY=$(kubectl get pod -n "${DR_NAMESPACE}" \ -l postgres-operator.crunchydata.com/cluster=dr,postgres-operator.crunchydata.com/role=primary \ -o jsonpath='{.items[0].metadata.name}') kubectl exec -n "${DR_NAMESPACE}" "${DR_PRIMARY}" -c database -- \ psql -t -A -c "SELECT pg_last_wal_replay_lsn() >= '<lsn>'::pg_lsn;"A repo-based standby replays WAL when it fetches a segment from the repository, so this step can take several seconds.
-
Promote the DR site:
kubectl patch pg dr -n "${DR_NAMESPACE}" --type=merge \ -p '{"spec":{"standby":{"enabled":false}}}' -
Confirm that the DR site accepts writes:
kubectl exec -n "${DR_NAMESPACE}" "${DR_PRIMARY}" -c database -- \ psql -t -A -c "SELECT pg_is_in_recovery();"The command returns
f. Switch your applications to the DR site. The Operator applies the user passwords about 10 seconds later. If your applications connect through pgBouncer, they can fail to log in for up to 30 seconds. -
Check the Main site. For a streaming standby,
patronictl listshows the Main member asStandby Leaderin thestreamingstate on the new timeline:kubectl exec -n "${MAIN_NAMESPACE}" "${MAIN_PRIMARY}" -c database -- patronictl listFor a repo-based standby, the member stays in
in archive recoveryon the old timeline. The PostgreSQL log repeatsnew timeline 2 forked off current database system timeline 1. Rebuild the Main site as described in the next section.
Rebuild the old primary of a repo-based standby¶
Choose one of these ways to bring the Main site back as a standby of DR.
Recreate the instance¶
This is the simplest way. After the swap, DR is the primary and holds all the writes that the Main site acknowledged, and the Main site is a standby. The data on the Main volumes has diverged from DR, so you can discard it. The Operator creates new instances, restores the latest backup from the pgBackRest repository, and replays the WAL from the repository until the Main site catches up with DR. Restore time depends on the size of your data.
Recreate every PostgreSQL instance of the Main site. The replicas of the old primary diverge as well, even though they report streaming and stay ready. If you recreate only the primary, Patroni makes a diverged replica the new standby leader.
Warning
Delete the Main volumes only after the swap is complete. Check that the DR site accepts writes (step 7), that the Main site is in recovery (step 4), and that the repository has a backup:
kubectl exec -n "${DR_NAMESPACE}" "${DR_PRIMARY}" -c database -- pgbackrest --stanza=db info
If you delete the volumes of a site that is still the only primary, or before DR has replayed the recorded position in step 5, you lose data.
-
Delete the PVCs and the Pods of all Main instances:
kubectl delete pvc -n "${MAIN_NAMESPACE}" \ -l postgres-operator.crunchydata.com/cluster=main,postgres-operator.crunchydata.com/data=postgres \ --wait=false kubectl delete pod -n "${MAIN_NAMESPACE}" \ -l postgres-operator.crunchydata.com/cluster=main,postgres-operator.crunchydata.com/data=postgres -
Check that every member is on the DR timeline:
kubectl exec -n "${MAIN_NAMESPACE}" <main-pod-name> -c database -- patronictl listAll members show the same
TLas DR: oneStandby Leaderand the replicas in thestreamingstate.
patronictl reinit doesn’t help here, because the Main site is a standby. On the primary it fails with No replica among provided members, and on the replicas it fails with invalid option '--scope=main-ha'.
Add a streaming source¶
With a streaming source, Patroni rewinds the Main site with pg_rewind and no restore is needed. The Main site must trust the DR certificates. This way is for a Main site with one PostgreSQL instance. If your site has replicas, recreate every instance instead.
-
Check that the certificates of the Main site and of the DR site are signed by the same CA, as in Configure DR site. The Main site then trusts the DR certificates, and it keeps its own certificates.
-
Set the DR
-haService as the standby host of the Main cluster:kubectl patch pg main -n "${MAIN_NAMESPACE}" --type=merge \ -p "{\"spec\":{\"standby\":{\"enabled\":true,\"repoName\":\"repo1\",\"host\":\"dr-ha.${DR_NAMESPACE}.svc\",\"port\":5432}}}"If the Main site doesn’t trust the DR certificates, the rewind fails with
SSL error: certificate verify failedand the Main site stays diverged.
Swap back¶
The roles are now reversed, and the swap back is the same procedure with the sites exchanged. For a repo-based standby, you rebuild the DR site the same way. To avoid the rebuild, add a streaming source to the standby before the next swap.
Re-apply the user passwords¶
Clear the verifier only if the passwords on the two sites differ. If you copied the user passwords and the pgBouncer password to both sites, and nobody changed them since, skip this section.
The Operator applies the passwords from the user Secrets of a cluster only when a Secret changes. A site that was primary before keeps the passwords that the other site set while it was primary, including a password that you changed there. The Operator doesn’t re-apply the pgBouncer user either, so pgBouncer on the promoted site can’t log in to PostgreSQL, unless both sites have the same pgBouncer password and verifier.
After you promote the site, clear the verifier in the Secret of each user, so that the Operator rebuilds it from the password in that Secret and applies it again. The example uses the Main site:
kubectl patch secret main-pguser-<user-name> -n "${MAIN_NAMESPACE}" \
-p '{"stringData":{"verifier":""}}'
The pgBouncer Secret works differently. If you copied the pgBouncer password and verifier between the sites, as in Copy the pgBouncer password, pgBouncer already works. Otherwise, clear the pgBouncer verifier. The Operator then generates a new pgBouncer password and a new verifier, and applies them:
kubectl patch secret main-pgbouncer -n "${MAIN_NAMESPACE}" \
-p '{"stringData":{"pgbouncer-verifier":""}}'
The user passwords from the Main Secrets apply again, and logins through pgBouncer work after about a minute. The pgBouncer passwords of the two sites differ afterwards, so copy the pgBouncer password and verifier again before the next swap.
To keep a user password that you changed on the other site, copy it to the Main Secret first, as in Prepare the user passwords. For the pgBouncer password, see Copy the pgBouncer password.
Created: November 18, 2026