Upgrade Database and Operator¶

Starting from the version 2.2.0 Percona Operator for PostgreSQL allows upgrades to newer 2.x versions.

Note

Upgrading from the 1.x branch of the Operator to 2.x versions ca be done in several ways and is completely different from the normal upgrade scenario due to substantial changes in the architecture.

Upgrading to a newer version typically involves two steps:

Upgrading the Operator and Custom Resource Definition (CRD) ,
Upgrading the Database Management System (Percona Distribution for PostgreSQL).

Alternatively, it is also possible to carry on minor version upgrades of Percona Distribution for PostgreSQL without the Operator upgrade.

Upgrading the Operator and CRD¶

The Operator version includes three numbers: major, minor, and patch (for example, the Operator version 2.6.0 has major version 2, minor version 6, and patch version 0). Only the incremental update to a nearest major.minor version of the Operator is supported. To update to a newer version, which differs from the current major.minor version by more than one, make several incremental updates sequentially.

For example, to upgrade the CRD and Operator from the version 2.4.0 to 2.6.0, the following sequence of upgrades will be the shortest recommended path:

upgrading the CRD and Operator from 2.4.0 to 2.5.1,
upgrading from 2.5.1 to 2.6.0.

You can find Operator versions listed here.

CRD supports last 3 minor versions of the Operator, which means it is compatible with the newest Operator version and the two older versions. If the Operator is older than the CRD by no more than two versions, you should be able to continue using the old Operator version. But updating the CRD and Operator is the recommended path.

Using newer CRD with older Operator is useful to upgrade multiple single-namespace Operator deployments in one Kubernetes cluster, where each Operator controls a database cluster in its own namespace. In this case upgrading Operator deployments will look as follows:

upgrade the CRD (not 3 minor versions far from the oldest Operator installation in the Kubernetes cluster) first
upgrade the Operators in each namespace incrementally to nearest minor version (e.g. first 2.4.0 to 2.5.1, then 2.5.1 to 2.6.0)

Manual upgrade¶

You can upgrade the Operator and CRD as follows, considering the Operator uses postgres-operator namespace, and you are upgrading to the version 2.6.0.

First update the CRD for the Operator, taking it from the official repository on Github (it is important to use --server-side flag when applying deploy/crd.yaml), and do the same for the Role-based access control. Applying the new CRD manifest must be done with server-side flag (otherwise you can encounter a number of errors caused by applying the CRD client-side: the command may fail, the built-in PosgreSQL extensions can be lost during such upgrade, etc.).
```
$ kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.6.0/deploy/crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.6.0/deploy/rbac.yaml -n postgres-operator
```
Note

In case of cluster-wide installation, use deploy/cw-rbac.yaml instead of deploy/rbac.yaml.
Now you should Apply a patch to your deployment, supplying necessary image name with a newer version tag. You can find the proper image names and version tags for the current Operator version in the list of certified images. For older versions, please refer to the old releases documentation archive ).

Updating to the 2.6.0 version should look as follows:
```
$ kubectl -n postgres-operator patch deployment percona-postgresql-operator \
   -p'{"spec":{"template":{"spec":{"containers":[{"name":"operator","image":"percona/percona-postgresql-operator:2.6.0"}]}}}}'
```
The deployment rollout will be automatically triggered by the applied patch. You can track the rollout process in real time with the kubectl rollout status command with the name of your cluster:
```
$ kubectl rollout status deployments percona-postgresql-operator -n postgres-operator
```
Expected output
```
deployment "percona-postgresql-operator" successfully rolled out
```

Upgrade via Helm¶

If you have installed the Operator using Helm, you can upgrade the Operator with the helm upgrade command.

Note

You can use helm upgrade to upgrade the Operator. But the database management system (Percona Distribution for PostgreSQL) should be upgraded in the same way whether you used Helm to install it or not.

Update the Custom Resource Definition for the Operator, taking it from the official repository on Github, and do the same for the Role-based access control:

$ kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.6.0/deploy/crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.6.0/deploy/rbac.yaml -n postgres-operator

If you installed the Operator with no customized parameters , the upgrade can be done as follows:
```
$ helm upgrade my-operator percona/pg-operator --version 2.6.0
```
The my-operator parameter in the above example is the name of a release object which which you have chosen for the Operator when installing its Helm chart.

If the Operator was installed with some customized parameters , you should list these options in the upgrade command.
Note

You can get list of used options in YAML format with the helm get values my-operator -a > my-values.yaml command, and this file can be directly passed to the upgrade command as follows:
```
$ helm upgrade my-operator percona/pg-operator --version 2.6.0 -f my-values.yaml
```

Upgrade via Operator Lifecycle Manager (OLM)¶

If you have installed the Operator on the OpenShift platform using OLM, you can upgrade the Operator within it.

List installed Operators for your Namespace to see if there are upgradable items.
Click the “Upgrade available” link to see upgrade details, then click “Preview InstallPlan” button, and finally “Approve” to upgrade the Operator.

Upgrading Percona Distribution for PostgreSQL¶

Before the Operator version 2.4, you could upgrade Percona Distribution for PostgreSQL from one minor version to another (such as upgrading from 15.5 to 15.7, or from 16.1 to 16.3). Starting from the Operator 2.4 you can also upgrade from one PostgreSQL major version to another (for example, upgrade from PostgreSQL 15.5 to PostgreSQL 16.3). Minor version upgrade and major version upgrade are technically different tasks with different scenarios.

Note

Upgrading a PostgreSQL cluster upgrade may result in downtime, as well as failover caused by updating the primary instance.

Minor version upgrade¶

Upgrading Percona Distribution for PostgreSQL minor version (for example, 16.1 to 16.3) can be done as follows:

Apply a patch to your Custom Resource, setting necessary Custom Resource version and image names with a newer version tag.

Note

Check the version of the Operator you have in your Kubernetes environment. Please refer to the Operator upgrade guide to upgrade the Operator and CRD first, if needed.

Patching Custom Resource is done with the kubectl patch pg command. Actual image names can be found in the list of certified images. For example, updating cluster1 cluster to the 2.6.0 version should look as follows:
```
$ kubectl -n postgres-operator patch pg cluster1 --type=merge --patch '{
   "spec": {
      "crVersion":"2.6.0",
      "image": "percona/percona-postgresql-operator:2.6.0-ppg17.4-postgres",
      "proxy": { "pgBouncer": { "image": "percona/percona-postgresql-operator:2.6.0-ppg17.4-pgbouncer1.24.0" } },
      "backups": { "pgbackrest":  { "image": "percona/percona-postgresql-operator:2.6.0-ppg17.4-pgbackrest2.54.2" } },
      "pmm": { "image": "percona/pmm-client:2.44.0" }
   }}'
```
The following image names in the above example were taken from the list of certified images:
- percona/percona-postgresql-operator:2.6.0-ppg17.4-postgres,
- percona/percona-postgresql-operator:2.6.0-ppg17.4-pgbouncer1.24.0,
- percona/percona-postgresql-operator:2.6.0-ppg17.4-pgbackrest2.54.2,
- percona/pmm-client:2.44.0.
Warning

The above command upgrades various components of the cluster including PMM Client. It is highly recommended to upgrade PMM Server before upgrading PMM Client. If it wasn’t done and you would like to avoid PMM Client upgrade, remove it from the list of images, reducing the last of two patch commands as follows:
```
$ kubectl -n postgres-operator patch pg cluster1 --type=merge --patch '{
   "spec": {
      "crVersion":"2.6.0",
      "image": "percona/percona-postgresql-operator:2.6.0-ppg17.4-postgres",
      "proxy": { "pgBouncer": { "image": "percona/percona-postgresql-operator:2.6.0-ppg17.4-pgbouncer1.24.0" } },
      "backups": { "pgbackrest":  { "image": "percona/percona-postgresql-operator:2.6.0-ppg17.4-pgbackrest2.54.2" } }
   }}'
```

The deployment rollout will be automatically triggered by the applied patch. The update process is successfully finished when all Pods have been restarted.

$ kubectl get pods -n postgres-operator

Expected output

NAME                                           READY   STATUS      RESTARTS   AGE
cluster1-backup-4vwt-p5d9j                     0/1     Completed   0          97m
cluster1-instance1-b5mr-0                      4/4     Running     0          99m
cluster1-instance1-b8p7-0                      4/4     Running     0          99m
cluster1-instance1-w7q2-0                      4/4     Running     0          99m
cluster1-pgbouncer-79bbf55c45-62xlk            2/2     Running     0          99m
cluster1-pgbouncer-79bbf55c45-9g4cb            2/2     Running     0          99m
cluster1-pgbouncer-79bbf55c45-9nrmd            2/2     Running     0          99m
cluster1-repo-host-0                           2/2     Running     0          99m
percona-postgresql-operator-79cd8586f5-2qzcs   1/1     Running     0          120m

Major version upgrade¶

Major version upgrade allows you to jump from one database major version to another (for example, upgrade from PostgreSQL 15.5 to PostgreSQL 16.3).

Note

Major version upgrades feature is currently a tech preview, and it is not recommended for production environments.

Also, currently the major version upgrade only works if the images in Custom Resource (deploy/cr.yaml manifest) are specified without minor version numbers:

...
image: percona/percona-postgresql-operator:2.4.0-ppg15-postgres
postgresVersion: 15
...

It will not work for images specified like percona/percona-postgresql-operator:2.4.0-ppg15.7-postgres.

The upgrade is triggered by applying the YAML file which refers to the special Operator upgrade image and contains the information about the existing and desired major versions. An example of this file is present in deploy/upgrade.yaml:

apiVersion: pgv2.percona.com/v2
kind: PerconaPGUpgrade
metadata:
  name: cluster1-15-to-16
spec:
  postgresClusterName: cluster1
  image: percona/percona-postgresql-operator:2.6.0-upgrade
  fromPostgresVersion: 15
  toPostgresVersion: 16
  toPostgresImage: percona/percona-postgresql-operator:2.6.0-ppg16.8-postgres
  toPgBouncerImage: percona/percona-postgresql-operator:2.6.0-ppg16.8-pgbouncer1.24.0
  toPgBackRestImage: percona/percona-postgresql-operator:2.6.0-ppg16.8-pgbackrest2.54.2

As you can see, the manifest includes image names for the database cluster components (PostgreSQL, pgBouncer, and pgBackRest). You can find them in the list of certified images for the current Operator release. For older versions, please refer to the old releases documentation archive ).

After you apply the YAML manifest as usual (by running kubectl apply -f deploy/upgrade.yaml command), the actual upgrade takes place:

The Operator pauses the cluster, so the cluster will be unavailable for the duration of the upgrade,
The cluster is specially annotated with pgv2.percona.com/allow-upgrade: <PerconaPGUpgrade.Name> annotation,
Jobs are created to migrate the data,
The cluster starts up after the upgrade finishes.

Note

If the upgrade fails for some reason, the cluster will stay in paused mode. Resume the cluster manually to check what went wrong with upgrade (it will start with the old version). You can check the PerconaPGUpgrade resource with kubectl get perconapgupgrade -o yaml command, and check the logs of the upgraded Pods to debug the issue.

During the upgrade data are duplicated in the same PVC for each major upgrade, and old version data are not deleted automatically. Make sure your PVC has enough free space to store data. You can remove data at your discretion by executing into containers and running the following commands (example for PostgreSQL 15):

$ rm -rf /pgdata/pg15
$ rm -rf /pgdata/pg15_wal

You can also delete the PerconaPGUpgrade resource (this will clean up the jobs and Pods created during the upgrade):

$ kubectl delete perconapgupgrade cluster1-15-to-16

Upgrading PostgreSQL extensions¶

If there are custom PostgreSQL extensions installed in the cluster, they need to be taken into account: you need to build and package each custom extension for the new PostgreSQL major version. During the upgrade the Operator will install extensions into the upgrade container.

The only built-in extension which demands special treatment after the database upgrade is pg_stat_monitor one. It is used to provide query analytics for Percona Monitoring and Management (PMM), if enabled in the Custom Resource (deploy/cr.yaml manifest):

extensions:
  ...
  builtin:
    pg_stat_monitor: true
    ...

If you need it, do the following after the database uprgade (this manual step is not required for the Operator versions 2.6.0 and newer):

Find the primary instance of your PostgreSQL cluster. You can do this using Kubernetes Labels as follows (replace the <namespace> placeholder with your value):

$ kubectl get pods -n <namespace> -l postgres-operator.crunchydata.com/cluster=cluster1 \ 
    -L postgres-operator.crunchydata.com/instance \
    -L postgres-operator.crunchydata.com/role | grep instance1

Sample output

cluster1-instance1-bmdp-0             4/4     Running   0          2m23s   cluster1-instance1-bmdp   replica
cluster1-instance1-fm7w-0             4/4     Running   0          2m22s   cluster1-instance1-fm7w   replica
cluster1-instance1-ttm9-0             4/4     Running   0          2m22s   cluster1-instance1-ttm9   master

PostgreSQL primary is labeled as master, while other PostgreSQL instances are labeled as replica.

Login to a primary instance (cluster1-instance1-ttm9-0 in the above example) as an administrative user:

$ kubectl exec  -n <namespace> -ti cluster1-instance1-ttm9-0 -c database -- psql postgres

Execute the following SQL statement:

postgres=# alter extension pg_stat_monitor update;

Upgrade from the Operator version 1.x to version 2.x¶

The Operator version 2.x has a lot of differences compared to the version 1.x. This makes upgrading from version 1.x to version 2.x quite different from a normal upgrade. In fact, you have to migrate the cluster from version 1.x to version 2.x.

There are several ways to do such version 1.x to version 2.x upgrade. Choose the method based on your downtime preference and roll back strategy:

	Pros	Cons
Data Volumes migration - re-use the volumes that were created by the Operator version 1.x	The simplest method	- Requires downtime - Impossible to roll back
Backup and restore - take the backup with the Operator version 1.x and restore it to the cluster deployed by the Operator version 2.x	Allows you to quickly test version 2.x	Provides significant downtime in case of migration
Replication - replicate the data from the Operator version 1.x cluster to the standby cluster deployed by the Operator version 2.x	- Quick test of v2 cluster - Minimal downtime during upgrade	Requires significant computing resources to run two clusters in parallel

Last update: 2025-04-07