Encrypt PXC traffic¶

Percona XtraDB Cluster processes two distinct types of traffic and applies encryption to ensure secure data exchanges across all connections.

Client-server traffic: flows between client applications and cluster nodes. This type of communication allows clients to send queries and retrieve data from the cluster, supporting standard database operations while maintaining data integrity and security.
Replication traffic: synchronizes replication traffic across cluster nodes using State Snapshot Transfer (SST), Incremental State Transfer (IST), write-set replication, and various service messages. These processes ensure data consistency and prevent loss.

Percona XtraDB Cluster encrypts all types of traffic to prevent unauthorized access and ensure data confidentiality. The system allows administrators to configure replication traffic encryption using either automated processes or manual setup, depending on security requirements and organizational policies. Implementing encryption enhances data protection, mitigates security risks, and maintains compliance with industry standards.

Encrypt client-server communication¶

SSL Encryption Framework in Percona XtraDB Cluster¶

Percona XtraDB Cluster uses MySQL’s native encryption framework to secure communication between client applications and cluster nodes. This ensures encrypted data exchange, prevents unauthorized access, and maintains end-to-end confidentiality.

Default Certificate Generation¶

Upon initialization, MySQL automatically generates default key and certificate files in the data directory (e.g. /var/lib/mysql). While these files simplify SSL setup, production environments should use custom certificates that meet internal security policies.
See: Generating Keys and Certificates Manually

Ensuring Consistency Across Nodes¶

All cluster nodes must use identical key and certificate files, which should be stored in /etc/ and configured in each node’s my.cnf. If the certificates differ between nodes, encrypted communication will fail, preventing nodes from joining the cluster.

To maintain a secure and unified cluster setup, you must:

Deploy identical SSL certificates to all nodes manually. This is the recommended approach to ensure secure communication.
Avoid disabling pxc_encrypt_cluster_traffic. Disabling this option does not contribute to a secure setup; instead, it exposes replication traffic to potential interception and eavesdropping, compromising the security of sensitive data.

A common practice is to generate the certificates on the first node and then copy them to all other nodes.

Recommended Certificate Storage¶

Instead of storing certificates in the data directory, relocate them to a dedicated directory:

mkdir -p /etc/mysql/certs
mv /var/lib/mysql/*.pem /etc/mysql/certs/
chown -R mysql:mysql /etc/mysql/certs

Then update your configuration in /etc/mysql/mysql.conf.d/mysqld.cnf:

ssl-key=/etc/mysql/certs/server-key.pem
ssl-cert=/etc/mysql/certs/server-cert.pem
ssl-ca=/etc/mysql/certs/ca.pem

The system reads the stored key and certificate files to secure client communication when the node restarts. These files encrypt data exchanges, ensuring privacy and protection. MySQL clients only need the second part of the configuration to connect to cluster nodes.

To ensure all nodes have identical certificates, it is recommended to generate key and certificate files on the first node. After generating the certificates, copy them to all other nodes to maintain consistency. This practice helps meet specific security policies and industry standards.

For detailed instructions on key creation, certificate signing, and configuration, refer to the section on Generate keys and certificates manually. This guide provides the necessary steps to set up encryption according to your security requirements.

Encrypt replication traffic¶

Replication traffic refers to the inter-node traffic, which includes the SST traffic, IST traffic, and replication traffic.

To ensure secure communication across the cluster, all nodes must use identical key and certificate files. Percona XtraDB Cluster provides a single configuration option to secure all replication traffic, known as SSL automatic configuration. This option ensures encrypted communication across the cluster.

Administrators can enhance security by configuring each channel separately using independent parameters, but it is crucial to maintain consistency in the certificates used across all nodes to prevent communication failures.

Use consistent SSL certificates across all cluster nodes¶

Automatic SSL encryption relies on key and certificate files to secure communication. MySQL generates default key and certificate files during initialization and stores them in the data directory.

All cluster nodes must use the same SSL certificates to maintain a secure and consistent encryption setup. Using different certificates on individual nodes can create connectivity problems and weaken security. Configure each node with identical SSL credentials to ensure reliable and encrypted communication across the cluster.

Configure SSL encryption¶

Percona XtraDB Cluster provides the pxc-encrypt-cluster-traffic variable to automate SSL encryption for replication traffic, including State Snapshot Transfer (SST), Incremental State Transfer (IST), and other replication processes. This configuration ensures that data exchanges between nodes remain encrypted, preventing unauthorized access and maintaining the cluster’s security.

By default, pxc-encrypt-cluster-traffic enables a secure communication channel for replication. This variable is not dynamic, meaning administrators cannot modify the value while running the cluster. Any change to the encryption settings requires stopping the cluster, updating the configuration, and restarting the nodes.

When enabled, pxc-encrypt-cluster-traffic automatically applies encryption settings, including encrypt, ssl_key, ssl-ca, and ssl-cert. The system uses these parameters to create a secure data exchange between nodes.

Administrators can explicitly configure encryption in my.cnf configuration file by setting pxc-encrypt-cluster-traffic=ON.

This configuration applies the following settings:

[mysqld]
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”

[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

For wsrep_provider_options, only the mentioned options are affected (socket.ssl_key, socket,ssl_cert, and socket.ssl_ca), the rest is not modified.

Disable pxc-encrypt-cluster-traffic and assess security risks¶

The default setting for PCX significantly enhances your system’s security by encrypting the traffic between nodes in a Percona XtraDB Cluster (PXC).

When the option for pxc-encrypt-cluster-traffic remains disabled, individuals with access to the network can connect to any PXC node, either as a client or as another node attempting to join the cluster. This unrestricted access allows these individuals to query sensitive data or obtain a complete copy of the data stored within the cluster.

To maintain a secure cluster, it is crucial to enable pxc-encrypt-cluster-traffic and ensure that all nodes use identical SSL certificates. If disabling pxc-encrypt-cluster-traffic becomes necessary, follow these steps to ensure proper configuration. First, halt the cluster to prevent any disruptions during the update process. Next, locate the [mysqld] section of the configuration file for each node and modify the setting to pxc-encrypt-cluster-traffic=OFF. After making this change, restart the cluster to apply the new configuration. This process ensures that the cluster operates under the updated security settings.

SSL automatic configuration¶

Automatic SSL encryption setup requires key and certificate files to secure communication. MySQL generates default key and certificate files and stores them in the data directory. These files support automatic SSL configuration, but all nodes must use the same key and certificate files to ensure a consistent security setup. Administrators can replace the auto-generated files with manually created certificates, as described in the Generate keys and certificates manually section.

When configuring encryption, the system first checks the SSL-ca, SSL, and SSL settings under [mysqld]. If the configuration does not include these options, the system actively searches the data directory for ca.pem, server-cert.pem, and server-key.pem. The system does not check the [sst] section for key and certificate files.

If the system locates all three required files, encryption is successfully configured. If any file is missing, the system generates a fatal error. Ensuring all necessary files are present prevents encryption failures and maintains secure communication within the cluster.

SSL manual configuration¶

A user who needs encryption for a specific channel or wants to use different certificates can configure encryption manually. This approach provides greater flexibility, allowing users to customize security settings based on operational requirements. However, it is essential that all nodes use identical key and certificate files to maintain a secure and consistent encryption setup.

To manually enable encryption, specify the locations of the required key and certificate files in the Percona XtraDB Cluster configuration. If the necessary files are unavailable, refer to Generate keys and certificates manually for instructions on creating them.

Encryption settings remain static during operation. Enabling encryption on a running cluster requires restarting the entire cluster.

You can enable encryption in three key areas of Percona XtraDB Cluster operation:

Encryption Type	Description
Encrypt SST traffic	Encrypts SST traffic during full data copy from one cluster node (donor) to the joining node (joiner).
Encrypt replication traffic	Encrypts all replication-related traffic within the cluster.
Encrypt IST traffic	Encrypts internal Percona XtraDB Cluster communication, including write-set replication, IST, and various service messages.

Encrypt SST traffic¶

When a new node (JOINER) joins the cluster, a full copy of the existing data is required to synchronize with the cluster. This process, known as State Snapshot Transfer (SST), ensures the new node receives a complete and accurate dataset from an existing node (DONOR). SST is a critical operation in Percona XtraDB Cluster because nodes start with a consistent state, avoiding data mismatches or inconsistencies.

To enable encryption during SST, administrators must configure the necessary settings in the cluster configuration file. When using the keyring_file plugin, encryption becomes mandatory. The system must transfer the keyring along with encrypted data files to ensure decryption. Without this step, the recipient node cannot access the transferred data.

For more information, see State snapshot transfer.

The following options are to be set in my.cnf on all nodes:

early-plugin-load=keyring_file.so
keyring-file-data=/path/to/keyring/file

The cluster will not work if keyring configuration across nodes is different. Additionally, all nodes must use identical key and certificate files to maintain a secure and consistent encryption setup.

xtrabackup¶

The only available SST method is xtrabackup-v2, which relies on Percona XtraBackup to transfer data securely between nodes. The wsrep_sst_method parameter is always set to xtrabackup-v2 and cannot be changed.

Encryption for this method is controlled by the encrypt option:

encrypt=0 is the default setting, meaning encryption is disabled.
encrypt=4 enables encryption using key and certificate files generated with OpenSSL.

For details on creating custom encryption keys and certificates, see Generating Keys and Certificates Manually

To enable encryption for SST using XtraBackup, define the paths to the key and certificate files in each node’s configuration under the [sst] section. This operation ensures secure data transfer between the donor and joiner nodes. Properly configured encryption prevents unauthorized access and maintains the integrity of the transferred data.

Note: All nodes must use identical key and certificate files to ensure a consistent security setup.

An example configuration in my.cnf:

[sst]
encrypt=4
ssl-ca=/etc/mysql/certs/ca.pem
ssl-cert=/etc/mysql/certs/server-cert.pem
ssl-key=/etc/mysql/certs/server-key.pem

SSL clients require DH parameters to be at least 1024 bits, due to the logjam vulnerability. However, versions of socat earlier than 1.7.3 use 512-bit parameters. If a dhparams.pem file of the required length is not found during SST in the data directory, a 2048-bit file is generated, which can take several minutes.

To avoid this delay, create the dhparams.pem file manually and place the file in the data directory before joining the node to the cluster.

An example command to generate the dhparams.pem file:

$ openssl dhparam -out /path/to/datadir/dhparams.pem 2048

For more information, see Percona XtraDB Cluster: “dh key too small” error during an SST using SSL.

Encrypt replication/IST traffic¶

Replication traffic in Percona XtraDB Cluster consists of the following key components:

Feature	Description
Write-set replication	The cluster continuously replicates transactions from one node to all other nodes, ensuring data consistency across the system.
Incremental State Transfer (IST)	Transfers only missing transactions from the donor node to the joining node, reducing overhead.
Service messages	Synchronization signals exchanged between nodes to maintain cluster consistency.

All replication traffic passes through the gcomm communication channel. Encrypting this channel secures IST traffic, write-set replication, and service messages ensuring confidentiality and integrity. IST uses a separate channel, so administrators must configure both channels with the same encryption parameters.

To enable encryption for all these processes, define the paths to the key, specify the paths to the key, certificate, and certificate authority files using the appropriate wsrep provider options:

To configure these encryption settings, define the required parameters in the configuration file using the wsrep_provider_options variable. This ensures secure replication traffic across the cluster.

wsrep_provider_options="socket.ssl=yes;socket.ssl_ca=/etc/mysql/certs/ca.pem;socket.ssl_cert=/etc/mysql/certs/server-cert.pem;socket.ssl_key=/etc/mysql/certs/server-key.pem"

All nodes must share the same key and certificate files to maintain a secure and consistent encryption setup. Ideally, use the same files configured for Encrypt client-server communication to simplify management and ensure compatibility across the cluster.

Check upgrade-certificate section on how to upgrade existing certificates.

Generate keys and certificates manually¶

The server automatically generates default key and certificate files and stores them in the data directory during initialization. Administrators can create new sets of key and certificate files to replace the default ones, tailoring security settings to specific encryption requirements.

Key and Certificate Type	Purpose
Certificate Authority (CA) key and certificate	Signs the server and client certificates to ensure authentication and trust.
Server key and certificate	Secures database server activity and encrypts write-set replication traffic within the cluster.
Client key and certificate	Encrypts client communication traffic to protect interactions with the database.

Generate the required key and certificate files using OpenSSL to ensure secure encryption.

The Common Name value assigned to the server and client keys and certificates must be unique and different from the value used for the Certificate Authority (CA) certificate. This distinction helps maintain proper authentication and prevents conflicts in identity verification.

Generate CA key and certificateGenerate server key and certificateGenerate client key and certificate

The Certificate Authority is used to verify the signature on certificates. These commands generate a Certificate Authority (CA) key and certificate:

Generate the CA key file:
```
$ openssl genrsa 2048 > ca-key.pem
```
The command generates a 2048-bit RSA private key and saves the key to ca-key.pem. This key is essential for signing certificates.
Generate the CA certificate file:
```
$ openssl req -new -x509 -nodes -days 3600
    -key ca-key.pem -out ca.pem
```
The command does the following:
- Generates a self-signed CA certificate valid for 3600 days.
- Uses the previously created private key (ca-key.pem).
- Ensures the private key is not encrypted with the -nodes flag.
- Outputs the certificate to ca.pem.

This CA certificate can then be used to sign other certificates for secure authentication within a system.

Generate a new RSA key pair and certificate request:
```
$ openssl req -newkey rsa:2048 -days 3600 \
    -nodes -keyout server-key.pem -out server-req.pem
```
The command does the following:
- Creates a 2048-bit RSA private key (server-key.pem).
- Generates a certificate signing request (CSR) (server-req.pem).
- -days 3600 sets the certificate validity period to 3600 days.
- -nodes ensures the private key remains unencrypted.
This command reads an RSA private key from the server-key.pem file, processes the key, and then writes the processed key back to the same file, removing the passphrase.
```
$ openssl rsa -in server-key.pem -out server-key.pem
```
This command generates a signed certificate from a CSR using a specified CA certificate and its corresponding private key. The command also sets a defined validity period and serial number for the new certificate.
```
$ openssl x509 -req -in server-req.pem -days 3600 \
    -CA ca.pem -CAkey ca-key.pem -set_serial 01 \
    -out server-cert.pem
```
The command does the following:
- Processes a certificate signing request (CSR) from the server-req.pem file.
- Sets the generated certificate’s validity period to 3600 days, using the -days 3600 option.
- Specifies the Certificate Authority (CA) certificate, ca.pem, to sign the CSR with the -CA ca.pem option.
- Provides the CA certificate’s private key, ca-key.pem, for signing the CSR using the -CAkey ca-key.pem option.
- Assigns a serial number of 01 to the newly created certificate using the -set_serial 01 option.
- Writes the resulting signed certificate to the server-cert.pem file, using the -out server-cert.pem option.

Generate the client key file:
```
$ openssl req -newkey rsa:2048 -days 3600 \
    -nodes -keyout client-key.pem -out client-req.pem
```
The command does the following:
- Creates a 2048-bit RSA private key (server-key.pem).
- Generates a certificate signing request (CSR) (server-req.pem).
- -days 3600 sets the certificate validity period to 3600 days.
- -nodes ensures the private key remains unencrypted.
Remove the passphrase:
```
$ openssl rsa -in client-key.pem -out client-key.pem
```
The command ensures the private key (server-key.pem) is stored without a passphrase.

Sign the certificate request using the CA certificate and key:

$ openssl x509 -req -in client-req.pem -days 3600 \
   -CA ca.pem -CAkey ca-key.pem -set_serial 01 \
   -out client-cert.pem

The command does the following:

Signs the CSR (server-req.pem) using the CA certificate (ca.pem) and CA key (ca-key.pem).
Generates a server certificate (server-cert.pem) valid for 3600 days.
Assigns a serial number (01) to the certificate.

Verify certificates¶

To check whether the server and client certificates are properly signed by the Certificate Authority (CA) certificate, run the following command:

$ openssl verify -CAfile ca.pem server-cert.pem client-cert.pem

This command verifies that the server and client certificates are valid and trusted by the specified CA certificate (ca.pem). If the certificates are correctly signed, OpenSSL returns a confirmation message;
otherwise, error details indicate issues with the certificate chain.

server-cert.pem: OK
client-cert.pem: OK

Failed validation caused by matching CN¶

An SSL configuration may fail if the certificate and CA files share the same attributes. To verify whether this is causing the issue, run the following openssl command and check that the CN field differs between the Subject and Issuer lines.

$ openssl x509 -in server-cert.pem -text -noout

Incorrect values

Certificate:
Data:
Version: 1 (0x0)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=www.percona.com, O=Database Performance., C=US
...
Subject: CN=www.percona.com, O=Database Performance., C=AU
...

To obtain a more compact output run openssl specifying -subject and -issuer parameters:

$ openssl x509 -in server-cert.pem -subject -issuer -noout

Expected output

subject= /CN=www.percona.com/O=Database Performance./C=AU
issuer= /CN=www.percona.com/O=Database Performance./C=US

Deploy keys and certificates¶

Use scp or sftp to send key and certificate files to each node securely.
Store files in /etc/mysql/certs/ or another accessible location.
Set permissions to limit access, ensuring only mysqld has read rights.

The following files are required:

File Type	Description
Certificate Authority certificate (`ca.pem`)	Verifies signatures to authenticate and establish trust.
Server key and certificate (`server-key.pem`, `server-cert.pem`)	Secures database server activity and encrypts write-set replication traffic.
Client key and certificate (`client-key.pem`, `client-cert.pem`)	Requires usage only if the node functions as a MySQL client, such as during SST using `mysqldump`.

Refer to the Upgrade certificates section for details on upgrading certificates when needed.

Upgrade certificates¶

The following procedure outlines the steps to upgrade certificates used for securing replication traffic when operating with two nodes in the cluster.

Restart the first node with the socket.ssl_ca option set to a combination of the the old and new certificates in a single file.

For example, you can merge contents of old-ca.pem and new-ca.pem into upgrade-ca.pem as follows:
```
cat old-ca.pem > upgrade-ca.pem && \
cat new-ca.pem >> upgrade-ca.pem
```
Set the wsrep_provider_options variable as follows:
```
wsrep_provider_options="socket.ssl=yes;socket.ssl_ca=/etc/mysql/certs/upgrade-ca.pem;socket.ssl_cert=/etc/mysql/certs/old-cert.pem;socket.ssl_key=/etc/mysql/certs/old-key.pem"
```

Restart the second node with the socket.ssl_ca, socket.ssl_cert, and socket.ssl_key options set to the corresponding new certificate files.

wsrep_provider_options="socket.ssl=yes;socket.ssl_ca=/etc/mysql/certs/new-ca.pem;socket.ssl_cert=/etc/mysql/certs/new-cert.pem;socket.ssl_key=/etc/mysql/certs/new-key.pem"

Restart the first node with the new certificate files, as in the previous step.
You can remove the old certificate files.