Skip to content
Percona Operators Documentation

Rate this page
☆ ☆ ☆ ☆ ☆
Thanks for your feedback
Thank you! The feedback has been submitted.

Get free database assistance or contact our experts for personalized support.

Get help from Percona

Configure data at rest encryption with TLS

This guide walks you through deploying and configuring HashiCorp Vault with TLS enabled to work with Percona Operator for MySQL based on Percona XtraDB Cluster to enable data-at-rest encryption using HTTPS protocol.

By default, Percona XtraDB Cluster (PXC) and Vault communicate over an unencrypted HTTP protocol. You can enable encrypted HTTPS protocol with TLS as an additional security layer to protect the data transmitted between Vault and your PXC nodes. HTTPS ensures that sensitive information, such as encryption keys and secrets, cannot be intercepted or tampered with on the network.

Assumptions

  1. This guide is provided as a best effort and builds upon procedures described in the official Vault documentation. Since Vault’s setup steps may change in future releases, this document may become outdated; we cannot guarantee ongoing accuracy or responsibility for such changes. For the most up-to-date and reliable information, please always refer to the official Vault documentation.
  2. For this setup we deploy the Vault server in High Availability (HA) mode on Kubernetes via Helm with TLS enabled. The HA setup uses Raft storage backend and consists of 3 replicas for redundancy. Using Helm is not mandatory. Any supported Vault deployment (on-premises, in the cloud, or a managed Vault service) works as long as the Operator can reach it.
  3. This guide uses Vault Helm chart version 0.30.0. You may want to change it to the required version by setting the VAULT_HELM_VERSION variable.

Prerequisites

Before you begin, ensure you have the following tools installed:

  • kubectl - Kubernetes command-line interface
  • helm - Helm package manager
  • openssl - OpenSSL toolkit for generating certificates
  • jq - JSON processor

Prepare your environment

  1. Export the namespace and other variables as environment variables to simplify further configuration:

    export NAMESPACE="vault"
export VAULT_HELM_VERSION="0.30.0"
export SERVICE="vault"
export CSR_NAME="vault-csr"
export SECRET_NAME_VAULT="vault-secret"
export POLICY_NAME="mysql-policy"
export WORKDIR="/tmp/vault"

  2. Create a working directory where for certificate and configuration files:

    mkdir -p /tmp/vault

  3. It is a good practice to isolate workloads in Kubernetes using namespaces. Create a namespace with the following command:

    kubectl create namespace vault

Generate certificates

To use TLS, you’ll need the following certificates:

  • A private key for the Vault server
  • A certificate for the Vault server signed by the Kubernetes CA
  • The Kubernetes CA certificate

These files store sensitive information. Make sure to keep them in a secure location.

Generate the private key

Generate a private key for the Vault server:

openssl genrsa -out ${WORKDIR}/vault.key 2048

Create the Certificate Signing Request (CSR)

A Certificate Signing Request (CSR) is a file that contains information about your server and the certificate you need. You create it using your private key, and then submit it to Kubernetes to get a certificate signed by the Kubernetes Certificate Authority (CA). The signed certificate proves your server’s identity and enables secure TLS connections.

  1. Create the Certificate Signing Request configuration file:

    Specify the certificate details that Kubernetes needs to sign your certificate:

    • Request settings ([req]): References the sections for certificate extensions and distinguished name. The distinguished name section is left empty as Kubernetes will populate it automatically.
    • Certificate extensions ([v3_req]): Defines how the certificate can be used. serverAuth allows the certificate for server authentication, while keyUsage specifies the cryptographic operations the certificate supports (non-repudiation, digital signature, and key encipherment).
    • Subject Alternative Names ([alt_names]): Lists all DNS names and IP addresses where your Vault service can be accessed. This includes the service name, fully qualified domain names (FQDNs) for different Kubernetes DNS contexts (namespace-scoped, cluster-scoped with .svc, and fully qualified with .svc.cluster.local), and the localhost IP address.
    cat > "${WORKDIR}/csr.conf" <<'EOF'
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.vault.svc.cluster.local
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 =*.vault-internal
DNS.2 = *.vault-standby
DNS.3 =*.vault-internal.vault.svc.cluster.local
DNS.4 = *.vault-standby.vault.svc.cluster.local
DNS.5 =*.vault
DNS.6 = vault.vault.svc.cluster.local
IP.1 = 127.0.0.1
EOF

  2. Generate the CSR. The following command creates the Certificate Signing Request file using your private key and the configuration file.

    The -subj parameter specifies the distinguished name directly: the Common Name (CN) identifies your Vault service using the Kubernetes node naming convention (system:node:${SERVICE}.${NAMESPACE}.svc), and the Organization (O) field is set to system:nodes, which Kubernetes requires to recognize and sign the certificate. The command combines these subject fields with the certificate extensions defined in the configuration file to produce the complete CSR.

    openssl req -new -key $WORKDIR/vault.key \
  -subj "/CN=system:node:${SERVICE}.${NAMESPACE}.svc;/O=system:nodes" \
  -out $WORKDIR/server.csr -config $WORKDIR/csr.conf

Issue the certificate

To get your certificate signed by Kubernetes, you need to submit the CSR through the Kubernetes API. The CSR file you generated with OpenSSL must be wrapped in a Kubernetes CertificateSigningRequest resource.

  1. Create the CSR YAML file to send it to Kubernetes:

    This YAML file creates a Kubernetes CertificateSigningRequest object that contains your CSR. The file embeds the base64-encoded CSR content and specifies:

    • The signer name (kubernetes.io/kubelet-serving) that tells Kubernetes which CA should sign the certificate
    • The groups field (system:authenticated) that identifies who can approve this CSR
    • The certificate usages that define how the certificate can be used (digital signature, key encipherment, and server authentication)
    cat > $WORKDIR/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: ${CSR_NAME}
spec:
  groups:
  - system:authenticated
  request: $(cat $WORKDIR/server.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kubelet-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

  2. Create the CertificateSigningRequest (CSR) object:

    kubectl create -f ${WORKDIR}/csr.yaml

  3. Approve the CSR in Kubernetes:

    kubectl certificate approve ${CSR_NAME}

  4. Confirm the certificate was issued:

    kubectl get csr ${CSR_NAME}
    Sample output 
    NAME        AGE   SIGNERNAME                      REQUESTOR       REQUESTEDDURATION   CONDITION
vault-csr   16s   kubernetes.io/kubelet-serving   minikube-user   <none>              Approved,Issued

Retrieve the certificates

After Kubernetes approves and signs your CSR, you need to retrieve the signed certificate and the Kubernetes CA certificate. These certificates are required to configure TLS for your Vault server.

  1. Retrieve the signed certificate from the CertificateSigningRequest object. The certificate is base64-encoded in Kubernetes, so you decode it and save it to a file.

    kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}' | base64 -d > $WORKDIR/vault.crt

  2. Retrieve Kubernetes CA certificate:

    This command retrieves the Kubernetes cluster’s Certificate Authority (CA) certificate from your kubeconfig file. The CA certificate is needed to verify that the signed certificate is valid and was issued by the Kubernetes CA. The command uses kubectl config view with flags to get the raw, flattened configuration and extract the CA certificate data, which is also base64-encoded.

    kubectl config view \
  --raw \
  --minify \
  --flatten \
  -o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
  | base64 -d > ${WORKDIR}/vault.ca

Store certificates in Kubernetes secrets

Create a TLS secret in Kubernetes to store the certificates and key:

kubectl create secret generic ${SECRET_NAME_VAULT} \
  --namespace ${NAMESPACE} \
  --from-file=vault.key=$WORKDIR/vault.key \
  --from-file=vault.crt=$WORKDIR/vault.crt \
  --from-file=vault.ca=$WORKDIR/vault.ca

Install Vault with TLS

For this setup, we install Vault in Kubernetes using the Helm 3 package manager in High Availability (HA) mode with Raft storage backend and with TLS enabled.

  1. Add and update the Vault Helm repository:

    helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

  2. Install Vault with TLS enabled:

    helm upgrade --install ${SERVICE} hashicorp/vault \
  --disable-openapi-validation \
  --version ${VAULT_HELM_VERSION} \
  --namespace ${NAMESPACE} \
  --set "global.enabled=true" \
  --set "global.tlsDisable=false" \
  --set "global.platform=kubernetes" \
  --set server.extraEnvironmentVars.VAULT_CACERT=/vault/userconfig/${SECRET_NAME_VAULT}/vault.ca \
  --set "server.extraEnvironmentVars.VAULT_TLSCERT=/vault/userconfig/${SECRET_NAME_VAULT}/vault.crt" \
  --set "server.extraEnvironmentVars.VAULT_TLSKEY=/vault/userconfig/${SECRET_NAME_VAULT}/vault.key" \
  --set "server.volumes[0].name=userconfig-${SECRET_NAME_VAULT}" \
  --set "server.volumes[0].secret.secretName=${SECRET_NAME_VAULT}" \
  --set "server.volumes[0].secret.defaultMode=420" \
  --set "server.volumeMounts[0].mountPath=/vault/userconfig/${SECRET_NAME_VAULT}" \
  --set "server.volumeMounts[0].name=userconfig-${SECRET_NAME_VAULT}" \
  --set "server.volumeMounts[0].readOnly=true" \
  --set "server.ha.enabled=true" \
  --set "server.ha.replicas=3" \
  --set "server.ha.raft.enabled=true" \
  --set "server.ha.raft.setNodeId=true" \
  --set-string "server.ha.raft.config=cluster_name = \"vault-integrated-storage\"
ui = true
listener \"tcp\" {
  tls_disable = 0
  address = \"[::]:8200\"
  cluster_address = \"[::]:8201\"
  tls_cert_file = \"/vault/userconfig/${SECRET_NAME_VAULT}/vault.crt\"
  tls_key_file  = \"/vault/userconfig/${SECRET_NAME_VAULT}/vault.key\"
  tls_client_ca_file = \"/vault/userconfig/${SECRET_NAME_VAULT}/vault.ca\"
}
storage \"raft\" {
  path = \"/vault/data\"
}
disable_mlock = true
service_registration \"kubernetes\" {}"

    This command does the following:

    • Installs HashiCorp Vault in High Availability (HA) mode with secure TLS enabled in your Kubernetes cluster.
    • Configures Vault pods to use certificates from a Kubernetes Secret via volume mounts for secure HTTPS communication between Vault and clients.
    • Sets up Raft as the backend storage with three replicas for fault tolerance, and configures the Vault TCP listener to enforce TLS with your specified certificate files.
    Sample output 
    NAME: vault
LAST DEPLOYED: Wed Aug 20 12:55:38 2025
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://developer.hashicorp.com/vault/docs

  3. Retrieve the Pod name where Vault is running:

    kubectl -n $NAMESPACE get pod -l app.kubernetes.io/name=${SERVICE} -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'
    Sample output 
    vault-0
vault-1
vault-2

Initialize and unseal Vault

  1. After Vault is installed, you need to initialize it. Run the following command to initialize the first pod:

    kubectl exec -it pod/vault-0 -n $NAMESPACE -- vault operator init -key-shares=1 -key-threshold=1 -format=json > ${WORKDIR}/vault-init

    The command does the following:

    • Connects to the Vault Pod
    • Initializes Vault server with TLS enabled
    • Creates 1 unseal key share which is required to unseal the server
    • Outputs the init response to a local file. The file includes unseal keys and root token.

  2. Vault is started in a sealed state. In this state Vault can access the storage but it cannot decrypt data. In order to use Vault, you need to unseal it.

    Retrieve the unseal key from the file:

    unsealKey=$(jq -r ".unseal_keys_b64[]" < ${WORKDIR}/vault-init)

    Now, unseal Vault. Run the following command:

    kubectl exec -it pod/vault-0 -n $NAMESPACE -- vault operator unseal "$unsealKey"
    Sample output 
    Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.19.0
Build Date              2025-03-04T12:36:40Z
Storage Type            raft
Cluster Name            vault-integrated-storage
Cluster ID              ed275c91-e227-681b-5aaa-f7a9fc19e37e
Removed From Cluster    false
HA Enabled              true
HA Cluster              <https://vault-0.vault-internal:8201>
HA Mode                 active
Active Since            2025-12-15T13:36:42.542059059Z
Raft Committed Index    37
Raft Applied Index      37

  3. Add the remaining Pods to the Vault cluster. If you have another secret name, replace the vault-secret with your value in the following for loop:

    for POD in vault-1 vault-2; do
  kubectl -n "$NAMESPACE" exec $POD -- sh -c '
    vault operator raft join -address=https://${HOSTNAME}.vault-internal:8200 \
      -leader-ca-cert="$(cat /vault/userconfig/vault-secret/vault.ca)" \
      -leader-client-cert="$(cat /vault/userconfig/vault-secret/vault.crt)" \
      -leader-client-key="$(cat /vault/userconfig/vault-secret/vault.key)" \
      https://vault-0.vault-internal:8200;
  '
done

    The command connects to each Vault Pod (vault-1 and vault-2) and issues the vault operator raft join command, which: * Joins the Pods to the Vault Raft cluster, enabling HA mode. * Uses the necessary TLS certificates to securely connect to the cluster leader (vault-0). * Ensures all nodes participate in the Raft consensus and share storage responsibilities.

    Sample output 
    Key       Value
---       -----
Joined    true

  4. Unseal the remaining Pods. Use this for loop:

    for POD in vault-1 vault-2; do
    kubectl -n "$NAMESPACE" exec $POD -- sh -c "
        vault operator unseal \"$unsealKey\"
    "
done
    Expected output 
    Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  true
Total Shares            1
Threshold               1
Unseal Progress         0/1
Unseal Nonce            n/a
Version                 1.19.0
Build Date              2025-03-04T12:36:40Z
Storage Type            raft
Removed From Cluster    false
HA Enabled              true

Configure Vault

At this step you need to configure Vault and enable secrets within it. To do so you must first authenticate in Vault.

When you started Vault, it generates and starts with a root token that provides full access to Vault. Use this token to authenticate.

Run the following command on a leader node. The remaining ones will synchronize from the leader.

  1. Extract the Vault root token from the file where you saved the init response output:

    cat ${WORKDIR}/vault-init | jq -r ".root_token"
    Sample output 
    hvs.*************Jg9r

  2. Connect to Vault Pod:

    kubectl exec -it vault-0 -n $NAMESPACE -- /bin/sh

  3. Authenticate in Vault with this token:

    vault login hvs.*************Jg9r

  4. Enable the secrets engine at the mount path. The following command enables KV secrets engine v2 at the secret mount-path:

    vault secrets enable --version=2 -path=pxc-secret kv
    Sample output 
    Success! Enabled the kv secrets engine at: secret/

  5. Optionally, enable audit logs for vault:

    vault audit enable file file_path=/vault/vault-audit.log

  6. Exit the Vault Pod:

    exit

Create a non-root token

Using the root token for authentication is not recommended, as it poses significant security risks. Instead, you should create a dedicated, non-root token for the Operator to use when accessing Vault. The permissions for this token are controlled by an access policy. Before you create a token you must first create the access policy.

  1. Create a policy for accessing the kv engine path and define the required permissions in the capabilities parameter:

    kubectl -n "$NAMESPACE" exec vault-0 -- sh -c "
  vault policy write $POLICY_NAME - <<EOF
path \"pxc-secret/data/*\" {
  capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]
}
path \"pxc-secret/metadata/*\" {
  capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]
}
path \"pxc-secret/*\" {
  capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]
}
EOF
"

  2. Now create a token with a policy.

    kubectl -n "${NAMESPACE}" exec pod/vault-0 -- vault token create -policy="${POLICY_NAME}" -format=json > "${WORKDIR}/vault-token.json

  3. Export the non-root token as an environment variable:

    export NEW_TOKEN=$(jq -r '.auth.client_token' "${WORKDIR}/vault-token.json")

  4. Verify the token:

    echo "New Vault Token: $NEW_TOKEN"
    Sample output 
    hvs.CAESINO******************************************T2Y

Create a Secret for Vault

To enable Vault for the Operator, create a Secret object for it. To do so, create a YAML configuration file and specify the following information:

  • A token to access Vault
  • A Vault server URL (using HTTPS)
  • The secrets mount path
  • Path to TLS certificates
  • Contents of the ca.cert certificate file. This is the vault.ca file in our example.

Depending on Percona XtraDB Cluster version, you must specify the Vault configuration as follows:

  • For Percona XtraDB Cluster 8.0 and 5.7 - as key=value pairs
  • For Percona XtraDB Cluster 8.4 - as a JSON object

You can modify the example configuration file:

apiVersion: v1
kind: Secret
metadata:
  name: keyring-secret-vault
type: Opaque
stringData:
  keyring_vault.conf: |-
    token = hvs.CAESINO******************************************T2Y
    vault_url = https://vault.vault.svc.cluster.local:8200
    secret_mount_point = secret
    vault_ca = /etc/mysql/vault-keyring-secret/ca.cert
  ca.cert: |-
    -----BEGIN CERTIFICATE-----
    MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
    EzARBgNVBAgTClNvbWUtU3RhdGUxFDASBgNVBAoTC0..0EgTHRkMTcwNQYD
    7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
    -----END CERTIFICATE-----

Replace the certificate content with the actual CA certificate from ${WORKDIR}/vault.ca.

apiVersion: v1
kind: Secret
metadata:
  name: keyring-secret-vault-84
type: Opaque
stringData:
  keyring_vault.conf: |-
    {
      "token": "hvs.CAESINO******************************************T2Y",
      "vault_url": "https://vault.vault.svc.cluster.local:8200",
      "secret_mount_point": "secret",
      "vault_ca": "/etc/mysql/vault-keyring-secret/ca.cert"
    }
  ca.cert: |-
    -----BEGIN CERTIFICATE-----
    MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
    EzARBgNVBAgTClNvbWUtU3RhdGUxFDASBgNVBAoTC0..0EgTHRkMTcwNQYD
    7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
    -----END CERTIFICATE-----

Replace the certificate content with the actual CA certificate from ${WORKDIR}/vault.ca.

Note

You must either specify the certificate value or don’t declare it at all. Having a commented #ca.cert field in the Secret configuration file is not allowed.

Now create a Secret object for your database cluster. Replace the <cluster-namespace> placeholder with the namespace where your database cluster is deployed:

kubectl apply -f deploy/vault-secret.yaml -n <cluster-namespace>

Reference the Secret in your Custom Resource manifest

Now, reference the Vault Secret in the Operator Custom Resource manifest. Note that the Secret name is the one you specified in the metadata.name field when you created a Secret.

  1. Export the namespace where the cluster is deployed and the secret name as environment variables:

    export CLUSTER_NAMESPACE=<cluster-namespace>
export SECRET_NAME_PXC="keyring-secret-vault-84"

  2. Update the cluster configuration. Since this is a running cluster, we will apply a patch referencing your Secret.

    Make sure to specify the correct Secret name. The default Secret name for MySQL 8.0 and 5.7 is keyring-secret-vault. In this setup we use keyring-secret-vault-84 Secret name for MySQL 8.4. Use the following command as an example and specify the Secret name for the MySQL version you’re using:

    kubectl patch pxc cluster1 \
  --namespace $CLUSTER_NAMESPACE \
  --type=merge \
  --patch "{\"spec\":{\"vaultSecretName\":\"${SECRET_NAME_PXC}\"}}"

This triggers cluster Pods to restart.

Use data-at-rest encryption

To use encryption, you can:

  • turn it on for every table you create with the ENCRYPTION='Y' clause in your SQL statement. For example:
CREATE TABLE t1 (c1 INT, PRIMARY KEY pk(c1)) ENCRYPTION='Y';
CREATE TABLESPACE foo ADD DATAFILE 'foo.ibd' ENCRYPTION='Y';

Existing tables will not be encrypted unless you specifically enable it via the ALTER TABLE .... ENCRYPTION='Y'; statement.

  • turn on default encryption of a schema or a general tablespace. Then all tables you create will have encryption enabled. To turn on default encryption, use the following SQL statement:
SET default_table_encryption=ON;

See the following chapters of Percona Server for MySQL documentation in how to use encryption:

Verify encryption

Refer to the Percona Server for MySQL documentation for guidelines how to verify encryption in your database.

Troubleshooting

If you encounter issues during the setup, use the following troubleshooting tips:

  1. Certificate Signing Request (CSR) issues: If you have problems with the CSR, manually delete it and recreate it:

    kubectl delete csr vault-csr || true

    Then recreate and re-approve it in Kubernetes following the steps in the Issue the certificate section.

  2. Vault policy issues: Check the mount points and permissions. Ensure that:

    • The mount path in your policy matches the path where you enabled the secrets engine
    • The policy has the required capabilities (create, read, update, delete, list) for the paths your application needs

  3. Mount point conflicts: If you encounter issues with a mount point in Vault, you cannot reuse it. You need to:

    • Provide a new mount path when enabling the secrets engine
    • Update your access policy to include the new mount path
    • Update the secret_mount_point value in your Secret configuration to match the new mount path

  4. Secret format mismatch: Check the MySQL version in your cluster and ensure you apply the correct format of the secret:

    • For Percona XtraDB Cluster 8.0 and 5.7 - use key=value pairs format
    • For Percona XtraDB Cluster 8.4 - use JSON object format

    Verify that the secret is created in the same namespace as your database cluster.

  5. Verify that you reference the correct secret name in your Custom Resource.

Clean up

After you finish working with Vault, you can clean up the temporary files:

rm -rf $WORKDIR
Last update: December 22, 2025
Created: December 22, 2025