PMM data encryption¶

Percona Monitoring and Management (PMM) encrypts the sensitive data it stores in its internal database, such as the access credentials and configuration details of monitored services.

Default encryption¶

PMM automatically manages encryption using a key file located at /srv/pmm-encryption.key. PMM generates this file upon the initial launch of PMM 3 or when upgrading from the latest version of PMM 2.

Use a custom encryption key¶

By default, PMM generates its own encryption key. To manage the key yourself, for example to keep it in your own secrets store, provide a custom key instead.

Important

Set up the custom key before PMM encrypts any data: before you upgrade to PMM 3, or before you start a new PMM 3 instance for the first time.

Key format¶

If you provide your own key, it must be in the format PMM expects: a base64-encoded Tink keyset created from the AES256GCMKeyTemplate. A raw 32-byte value doesn’t work, so don’t create the key with a general-purpose tool such as openssl rand. If PMM can’t read the key file, it doesn’t start.

To get a key in the right format, generate it with the Encryption Rotation Tool.

Set up a custom key¶

To set up a custom key:

Generate the key with pmm-encryption-rotation, a tool that comes with PMM Server, so there’s nothing to install. Run it from the PMM Server Docker image:
```
docker run --rm --entrypoint /usr/sbin/pmm-encryption-rotation percona/pmm-server:3 --generate-key > pmm-encryption.key
```
This starts a temporary container, saves a new key to pmm-encryption.key in your current directory, and removes the container. The command doesn’t connect to any database, so you can run it before you start PMM Server.

If PMM Server is already running, you can instead run pmm-encryption-rotation --generate-key > pmm-encryption.key inside its container.
Make the key file available inside the PMM Server container, for example on a mounted volume.
Set the PMM_ENCRYPTION_KEY_PATH environment variable to the path of the key file inside the container.

Keep the key safe¶

PMM uses the custom key to encrypt and decrypt all credentials it stores. If you lose the key, PMM can’t decrypt those credentials. To avoid this:

store the key securely and keep a backup of it outside PMM.
in containerized environments, set PMM_ENCRYPTION_KEY_PATH in the container configuration so it persists across restarts.
test key rotation in a staging environment before you rotate the key in production.

HA clusters: use the same key on every node¶

In a high availability (HA) cluster, all PMM Server nodes share one PostgreSQL database, but each node reads the encryption key from its own local file. This means every node must use the same key.

A node with a different key can’t decrypt the credentials that other nodes stored. PMM detects the mismatch and stops that node from starting. Otherwise, the node would send unusable credentials to PMM Clients, and monitoring would stop for those services.

The PMM HA Helm chart manages the shared key for you, including across upgrades and rescaling.

If you set up HA nodes yourself with the HA variables:

Generate a single key for the whole cluster.
Copy the key file to /srv/pmm-encryption.key on every node, or to the path set in PMM_ENCRYPTION_KEY_PATH.
Back up the key file together with the rest of your cluster configuration.
Start the nodes.

If a node has a different key, it logs encryption key does not match the database and doesn’t start. To fix it, copy the key file from a node that works to the same path on that node, then restart the node.

Rotate the encryption key¶

Rotate the encryption key when it is compromised or as part of your regular security maintenance. The pmm-encryption-rotation tool generates a new key and re-encrypts all stored credentials with it.

PMM Server stops briefly and restarts during rotation. Plan for a short monitoring gap.

Prepare a custom setup¶

Skip this section if you use the default key path and default database settings.

Custom key path¶

Make sure PMM_ENCRYPTION_KEY_PATH points to the current key, so the tool can decrypt existing data.
Make sure the pmm user (UID 1000) can write to the directory that contains the key file. Otherwise, the rotation fails with a permission denied error and leaves pmm-managed stopped. To fix this, see Recover from a failed rotation.

Custom database credentials or SSL¶

Have the credentials or SSL settings for the PMM internal database ready to pass as options to the rotation tool. To list the available options, run pmm-encryption-rotation --help.

Rotate the key¶

To rotate the key:

Log in to the container that runs PMM Server.
Run the rotation tool. If you use custom database credentials or SSL, add them as options:
```
pmm-encryption-rotation
```
The tool saves the new key to the same location as the old one: /srv/pmm-encryption.key, or the path set in PMM_ENCRYPTION_KEY_PATH. The previous key is saved as pmm-encryption_old.key in the same directory.
If you installed PMM Server with the Helm chart, restart the pod so the chart updates its backup copy of the key:
```
kubectl delete pod pmm-0
```
Confirm that PMM is working: check that your services appear as connected in Inventory > Services and that metrics are visible in your dashboards.
Delete pmm-encryption_old.key or move it to secure offline storage. This file contains the previous key in plain text.

Rotate the key on HA nodes¶

Key rotation isn’t supported yet for the PMM HA Helm chart. The chart mounts the key from a read-only Kubernetes secret, which the rotation tool can’t replace.

If you set up HA nodes yourself with the HA variables, all nodes must switch to the new key together. Rotate it on one node, then copy it to the others:

Stop PMM Server on every node except one. Any node left running keeps using the old key, which leaves the database encrypted with two different keys.
On the remaining node, rotate the encryption key.
Copy the new key file from that node to every other node, using the same path.
Start the other nodes.

Recover from a failed rotation¶

If PMM Server stops while pmm-encryption-rotation is running, your data remains intact and encrypted with the previous key.

To bring PMM back up and retry:

Restart pmm-managed:
```
supervisorctl start pmm-managed
```
If the directory containing your key file is not writable by the pmm user (UID 1000), fix the permissions:
```
chown pmm:pmm /path/to/key/directory
```
Run the rotation tool again.

Recover from a corrupted rotation¶

PMM versions before 3.9.1 contained a bug that corrupted certain credentials during key rotation. If you rotated the encryption key before upgrading to 3.9.1, see Corrupted credentials after encryption key rotation for recovery steps.