Update Database and Operator¶
Starting from the version 1.1.0 the Percona Operator for MongoDB allows upgrades to newer versions. The upgradable components of the cluster are the following ones:
- the Operator;
- Custom Resource Definition (CRD),
- Database Management System (Percona Server for MongoDB).
The list of recommended upgrade scenarios includes two variants:
- Upgrade to the new versions of the Operator and Percona Server for MongoDB,
- Minor Percona Server for MongoDB version upgrade without the Operator upgrade.
Upgrading the Operator and CRD¶
Note
The Operator supports last 3 versions of the CRD, so it is technically possible to skip upgrading the CRD and just upgrade the Operator. If the CRD is older than the new Operator version by no more than three releases, you will be able to continue using the old CRD and even carry on Percona Server for MongoDB minor version upgrades with it. But the recommended way is to update the Operator and CRD.
Only the incremental update to a nearest version of the Operator is supported (for example, update from 1.5.0 to 1.6.0). To update to a newer version, which differs from the current version by more than one, make several incremental updates sequentially.
Note
Starting from version 1.14.0, the Operator configures replica set members
using local fully-qualified domain names (FQDN). Before this version, it
used exposed IP addresses in the replica set configuration in case of the
exposed replica set.
If you have your replica set exposed and upgrade to 1.14.0, the
replica set configuration will change to use FQDN.
If you don’t want such reconfiguration to happen, set
clusterServiceDNSMode
Custom Resource option to External
before the
upgrade.
Manual upgrade¶
The upgrade includes the following steps.
-
Update the Custom Resource Definition for the Operator, taking it from the official repository on Github, and do the same for the Role-based access control:
$ kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.14.0/deploy/crd.yaml $ kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.14.0/deploy/rbac.yaml
-
Now you should apply a patch to your deployment, supplying necessary image name with a newer version tag. You can find the proper image name for the current Operator release in the list of certified images. updating to the
1.14.0
version should look as follows:$ kubectl patch deployment percona-server-mongodb-operator \ -p'{"spec":{"template":{"spec":{"containers":[{"name":"percona-server-mongodb-operator","image":"percona/percona-server-mongodb-operator:1.14.0"}]}}}}'
-
The deployment rollout will be automatically triggered by the applied patch. You can track the rollout process in real time with the
kubectl rollout status
command with the name of your cluster:$ kubectl rollout status deployments percona-server-mongodb-operator
Note
Labels set on the Operator Pod will not be updated during upgrade.
Upgrade via helm¶
If you have installed the Operator using Helm, you can upgrade the
Operator with the helm upgrade
command.
-
In case if you installed the Operator with no customized parameters, the upgrade can be done as follows:
$ helm upgrade my-op percona/psmdb-operator --version 1.14.0
The
my-op
parameter in the above example is the name of a release object which which you have chosen for the Operator when installing its Helm chart.If the Operator was installed with some customized parameters, you should list these options in the upgrade command.
Note
You can get list of used options in YAML format with the
helm get values my-op -a > my-values.yaml
command, and this file can be directly passed to the upgrade command as follows:$ helm upgrade my-op percona/psmdb-operator --version 1.14.0 -f my-values.yaml
-
Update the Custom Resource Definition for the Operator, taking it from the official repository on Github, and do the same for the Role-based access control:
$ kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.14.0/deploy/crd.yaml $ kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.14.0/deploy/rbac.yaml
Note
You can use helm upgrade
to upgrade the Operator only. The Database (Percona Server for MongoDB) should be upgraded in the same way whether you used helm to install it or not.
Upgrading Percona Server for MongoDB¶
The following section presumes that you are upgrading your cluster within the
Smart Update strategy, when the Operator controls how the objects
are updated. Smart Update strategy is on when the updateStrategy
key in the
Custom Resource configuration file is set to SmartUpdate
(this is the default value and the recommended way for upgrades).
Note
As an alternative, the updateStrategy
key can be used to turn off
Smart Update strategy. You can find out more on this in the
appropriate section.
Manual upgrade¶
Manual update of Percona Server for MongoDB can be done as follows:
-
Make sure that
spec.updateStrategy
option in the Custom Resource is set toSmartUpdate
,spec.upgradeOptions.apply
option is set toNever
orDisabled
(this means that the Operator will not carry on upgrades automatically).... spec: updateStrategy: SmartUpdate upgradeOptions: apply: Disabled ...
-
Now apply a patch to your Custom Resource, setting necessary Custom Resource version and image names with a newer version tag.
Note
Check the version of the Operator you have in your Kubernetes environment. Please refer to the Operator upgrade guide to upgrade the Operator and CRD, if needed.
Patching Custom Resource is done with the
kubectl patch psmdb
command. Actual image names can be found in the list of certified images. For example, updatingmy-cluster-name
cluster to the1.14.0
version should look as follows:$ kubectl patch psmdb my-cluster-name --type=merge --patch '{ "spec": { "crVersion":"1.14.0", "image": "percona/percona-server-mongodb:4.4.18-18", "backup": { "image": "percona/percona-backup-mongodb:2.0.4" }, "pmm": { "image": "percona/pmm-client:2.35.0" } }}'
Warning
The above command upgrades various components of the cluster including PMM Client. It is highly recommended to upgrade PMM Server before upgrading PMM Client. If it wasn’t done and you would like to avoid PMM Client upgrade, remove it from the list of images, reducing the last of two patch commands as follows:
$ kubectl patch psmdb my-cluster-name --type=merge --patch '{ "spec": { "crVersion":"1.14.0", "image": "percona/percona-server-mongodb:4.4.18-18", "backup": { "image": "percona/percona-backup-mongodb:2.0.4" } }}'
-
The deployment rollout will be automatically triggered by the applied patch. You can track the rollout process in real time using the
kubectl rollout status
command with the name of your cluster:$ kubectl rollout status sts my-cluster-name-rs0
The update process is successfully finished when all Pods have been restarted (including the mongos and Config Server nodes, if Percona Server for MongoDB Sharding is on).
Automated upgrade¶
Smart Update strategy allows you to automate upgrades even more. In this case the Operator can either detect the availability of the new Percona Server for MongoDB version, or rely on the user’s choice of the version. To check the availability of the new version, the Operator will query a special Version Service server at scheduled times to obtain fresh information about version numbers and valid image paths.
If the current version should be upgraded, the Operator updates the Custom
Resource to reflect the new image paths and carries on sequential Pods deletion,
allowing StatefulSet to redeploy the cluster Pods with the new image.
You can configure Percona Server for MongoDB upgrade via the deploy/cr.yaml
configuration file as follows:
-
Make sure that
spec.updateStrategy
option is set toSmartUpdate
. -
Change
spec.crVersion
option to match the version of the Custom Resource Definition upgrade you have done while upgrading the Operator:... spec: crVersion: 1.14.0 ...
Note
If you don’t update crVersion, minor version upgrade is the only one to occur. For example, the image
percona-server-mongodb:5.0.7-6
can be upgraded topercona-server-mongodb:5.0.11-10
. -
Set the
upgradeOptions.apply
option fromDisabled
to one of the following values:Recommended
- automatic upgrade will choose the most recent version of software flagged as Recommended (for clusters created from scratch, the Percona Server for MongoDB 6.0 version will be selected instead of the Percona Server for MongoDB 5.0 or 4.4 version regardless of the image path; for already existing clusters, the 6.0 vs. 5.0 or 4.4 branch choice will be preserved),6.0-recommended
,5.0-recommended
,4.4-recommended
- same as above, but preserves specific major MongoDB version for newly provisioned clusters (ex. 6.0 will not be automatically used instead of 5.0),Latest
- automatic upgrade will choose the most recent version of the software available (for clusters created from scratch, the Percona Server for MongoDB 6.0 version will be selected instead of the Percona Server for MongoDB 5.0 or 4.4 version regardless of the image path; for already existing clusters, the 6.0 vs. 5.0 or 4.4 branch choice will be preserved),6.0-latest
,5.0-latest
,4.4-latest
- same as above, but preserves specific major MongoDB version for newly provisioned clusters (ex. 6.0 will not be automatically used instead of 5.0),- version number - specify the desired version explicitly (version numbers are specified as 6.0.4-3, 5.0.14-12, etc.). Actual versions can be found in the list of certified images.
-
Make sure the
versionServiceEndpoint
key is set to a valid Version Server URL (otherwise Smart Updates will not occur).You can use the URL of the official Percona’s Version Service (default). Set
upgradeOptions.versionServiceEndpoint
tohttps://check.percona.com
.Alternatively, you can run Version Service inside your cluster. This can be done with the
kubectl
command as follows:$ kubectl run version-service --image=perconalab/version-service --env="SERVE_HTTP=true" --port 11000 --expose
Note
Version Service is never checked if automatic updates are disabled in the
upgradeOptions.apply
option. If automatic updates are enabled, but the Version Service URL can not be reached, no updgrades will be performed. -
Use the
upgradeOptions.schedule
option to specify the update check time in CRON format.The following example sets the midnight update checks with the official Percona’s Version Service:
spec: updateStrategy: SmartUpdate upgradeOptions: apply: Recommended versionServiceEndpoint: https://check.percona.com schedule: "0 0 * * *" ...
Note
You can force an immediate upgrade by changing the schedule to
* * * * *
(continuously check and upgrade) and changing it back to another more conservative schedule when the upgrade is complete. -
Don’t forget to apply your changes to the Custom Resource in the usual way:
$ kubectl apply -f deploy/cr.yaml
Note
When automatic upgrades are disabled by the
apply
option, Smart Update functionality will continue working for changes triggered by other events, such as rotating a password, or changing resource values.
Major version automated upgrades¶
Normally automatic upgrade takes place within minor versions (for example,
from 4.4.16-16
to 4.4.18-18
) of MongoDB. Major versions upgrade (for
example moving from 5.0-recommended
to 6.0-recommended
) is more
complicated task which might potentially affect how data is stored and how
applications interacts with the database (in case of some API changes).
Such upgrade is supported by the Operator within one major version at a time: for example, to change Percona Server for MongoDB major version from 4.4 to 6.0, you should first upgrade it to 5.0, and later make a separate upgrade from 5.0 to 6.0. The same is true for major version downgrades.
Note
It is recommended to take a backup before upgrade, as well as to perform upgrade on staging environment.
Major version upgrade can be initiated using the upgradeOptions.apply
key in the deploy/cr.yaml
configuration file:
spec:
upgradeOptions:
apply: 5.0-recommended
Note
When making downgrades (e.g. changing version from 5.0 to 4.4), make sure to remove incompatible features that are persisted and/or update incompatible configuration settings. Compatibility issues between major MongoDB versions can be found in upstream documentation.
By default the Operator doesn’t set
FeatureCompatibilityVersion (FCV)
to match the new version, thus making sure that backwards-incompatible features
are not automatically enabled with the major version upgrade (which is
recommended and safe behavior). You can turn this backward compatibility off at
any moment (after the upgrade or even before it) by setting the
upgradeOptions.setFCV flag in the
deploy/cr.yaml
configuration file to true
.
Note
With setFeatureCompatibilityVersion set major version rollback is not
currently supported by the Operator. Therefore it is recommended to stay
without enabling this flag for some time after the major upgrade to ensure
the likelihood of downgrade is minimal. Setting setFCV
flag to true
simultaneously with the apply
flag should be done only if the whole
procedure is tested on staging and you are 100% sure about it.
More on upgrade strategies¶
The recommended way to upgrade your cluster is to use the
Smart Update strategy, when the Operator controls how the objects
are updated. Smart Update strategy is on when the updateStrategy
key in the
Custom Resource configuration file is set to SmartUpdate
(this is the default value and the recommended way for upgrades).
Alternatively, you can set this key to RollingUpdate
or OnDelete
, which
means that you will have to
follow the low-level Kubernetes way of database upgrades.
But take into account, that SmartUpdate
strategy is not just for simplifying
upgrades. Being turned on, it allows to disable automatic upgrades, and still
controls restarting Pods in a proper order for changes triggered by other
events, such as updating a ConfigMap, rotating a password, or changing resource
values. That’s why SmartUpdate
strategy is useful even when you have no plans
to automate upgrades at all.