Transport layer security (TLS)¶

The Percona Operator for PostgreSQL uses Transport Layer Security (TLS) cryptographic protocol for the following types of communication:

Internal - communication between PostgreSQL instances in the cluster
External - communication between the client application and the cluster

The internal certificate is also used as an authorization method for PostgreSQL Replica instances.

TLS security can be configured in following ways:

the Operator can generate long-term certificates automatically at cluster creation time,
you can generate certificates manually.

The following subsections explain how to configure TLS security with the Operator yourself, as well as how to temporarily disable it if needed.

Allow the Operator to generate certificates automatically¶

The Operator is able to generate long-term certificates automatically and turn on encryption at cluster creation time, if there are no certificate secrets available. Just deploy your cluster as usual, with the kubectl apply -f deploy/cr.yaml command, and certificates will be generated.

Note

With the Operator versions before 2.5.0, autogenerated certificates for all database clusters were based on the same generated root CA. Starting from 2.5.0, the Operator creates root CA on per-cluster basis.

Check connectivity to the cluster¶

You can check TLS communication with use of the psql, the standard interactive terminal-based frontend to PostgreSQL. The following command will spawn a new pg-client container, which includes needed command and can be used for the check (use your real cluster name instead of the <cluster-name> placeholder):

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pg-client
spec:
  replicas: 1
  selector:
    matchLabels:
      name: pg-client
  template:
    metadata:
      labels:
        name: pg-client
    spec:
      containers:
        - name: pg-client
          image: perconalab/percona-distribution-postgresql:16.4
          imagePullPolicy: Always
          command:
          - sleep
          args:
          - "100500"
          volumeMounts:
            - name: ca
              mountPath: "/tmp/tls"
      volumes:
      - name: ca
        secret:
          secretName: <cluster_name>-ssl-ca
          items:
          - key: ca.crt
            path: ca.crt
            mode: 0777
EOF

Now get shell access to the newly created container, and launch the PostgreSQL interactive terminal to check connectivity over the encrypted channel (please use real cluster-name, PostgreSQL user login and password):

$ kubectl exec -it deployment/pg-client -- bash -il
[postgres@pg-client /]$ PGSSLMODE=verify-ca PGSSLROOTCERT=/tmp/tls/ca.crt psql postgres://<postgresql-user>:<postgresql-password>@<cluster-name>-pgbouncer.<namespace>.svc.cluster.local

Now you should see the prompt of PostgreSQL interactive terminal:

$ psql (16.4)
Type "help" for help.
pgdb=>

Generate certificates manually¶

Provide pre-existing certificates to the Operator¶

To allow the Operator to use custom certificates, simply create the appropriate Secrets in your cluster namespace before deploying the cluster with the kubectl apply -f deploy/cr.yaml command. The Secret should contain the TLS key (tls.key), TLS certificate (tls.crt) and the CA certificate (ca.crt) to use:

apiVersion: v1
kind: Secret
metadata:
  name: cluster1-cert
type: Opaque
data:
  ca.crt: <value>
  tls.crt: <value>
  tls.key: <value>

For example, if you have files named ca.crt, my_tls.key, and my_tls.crt stored on your local machine, you could run the following command to create a Secret named cluster1.tls in the postgres-operator namespace:

$ kubectl create secret generic -n postgres-operator cluster1.tls \
  --from-file=ca.crt=ca.crt \
  --from-file=tls.key=my_tls.key \
  --from-file=tls.crt=my_tls.crt

You should use two sets of certificates: one set is for external communications, and another set is for internal ones. A secret created for the external use must be added to the secrets.customTLSSecret.name field of your Custom Resource. A certificate generated for internal communications must be added to the secrets.customReplicationTLSSecret.name field in your Custom Resource. You can do it in the deplou/cr.yaml configuration file as follows:

spec:
  ...
  secrets:
     customTLSSecret:
       name: cluster1-cert
     customReplicationTLSSecret:
       name: replication1-cert
  ...

Don’t forget to apply changes as usual:

$ kubectl apply -f deploy/cr.yaml

Provide custom root CA certificate to the Operator¶

You can also provide a custom root CA certificate to the Operator. In this case the Operator will not generate one itself, but will use the user-provided CA. Particularly, this can be useful if you would like to have several database clusters with certificates generated by the Operator based on the same root CA.

To make the Operator using custom root certificate, create a separate secret with this certificate and specify this secret in Custom Resource options.

For example, if you have files named my_tls.key and my_tls.crt stored on your local machine, you could run the following command to create a Secret named cluster1-ca-cert in the postgres-operator namespace:

$ kubectl create secret generic -n postgres-operator cluster1-ca-cert \
  --from-file=tls.crt=my_tls.crt \
  --from-file=tls.key=my_tls.key

You also need to specify details about this secret in your deploy/cr.yaml manifest:

...
secrets:
  customRootCATLSSecret:
    name: cluster1-ca-cert
    items:
      - key: "tls.crt"
        path: "root.crt"
      - key: "tls.key"
        path: "root.key"

Now, when you create the cluster with the kubectl apply -f deploy/cr.yaml command, the Operator should use the root CA certificate you had provided.

Warning

This approach allows using root CA certificate auto-generated by the Operator for some other clusters, but it needs caution. If the cluster with auto-generated certificate has delete-ssl finalizer enabled, the certificate will be deleted at the cluster deletion event even if it was manually provided to some other cluster.

Generate custom certificates for the Operator yourself¶

The good option to find out the certificates specifics needed for the Operator would be to look at certificates, generated by the Operator automatically. Supposing that your cluster name is cluster1, you can examine the auto-generated CA certificate (ca.crt) after deploying the cluster as follows:

$ kubectl get secret/cluster1-cluster-cert -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -text -noout

Expected output

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ec:f3:d6:f5:35:5c:97:0c:66:cc:90:ed:e6:4b:0a:07
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: CN = postgres-operator-ca
        Validity
            Not Before: Dec 24 13:58:21 2023 GMT
            Not After : Dec 21 14:58:21 2033 GMT
        Subject: CN = postgres-operator-ca
        Subject Public Key Info:
        ...
    ...

You can check the auto-generated TLS certificate (tls.crt) in a similar way:

$ kubectl get secret/cluster1-cluster-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout

Expected output

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            59:f3:44:09:f1:73:b3:8e:ba:d4:a0:52:cc:fb:9c:1f
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: CN = postgres-operator-ca
        Validity
            Not Before: Dec 24 13:58:21 2023 GMT
            Not After : Dec 23 14:58:21 2024 GMT
        Subject: CN = cluster1-primary.default.svc.cluster.local.
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:b1:2f:37:1b:ca:ab:5f:19:38:24:69:11:54:82:
                    10:49:fd:00:3c:26:ef:83:32:82:b1:73:96:e8:9d:
                    eb:2f:60:89:ea:3a:cb:95:a7:0a:2e:46:63:ce:29:
                    87:17:1a:d4:3e:c5:5a:90:8c:71:3b:23:75:21:42:
                    09:60:81:da:c1
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                3C:25:65:88:F2:CD:29:37:05:06:7C:E8:F3:C4:2B:CD:9B:DC:5E:74
            X509v3 Subject Alternative Name: 
                DNS:cluster1-primary.default.svc.cluster.local., DNS:cluster1-primary.default.svc, DNS:cluster1-primary.default, DNS:cluster1-primary, DNS:cluster1-replicas.default.svc.cluster.local., DNS:cluster1-replicas.default.svc, DNS:cluster1-replicas.default, DNS:cluster1-replicas
    Signature Algorithm: ecdsa-with-SHA384
    ...

While sharing the same ca.crt, certificates for external communications (referenced in the secrets.customTLSSecret.name Custom Resource option) and certificates for internal ones (referenced in the secrets.customReplicationTLSSecret.name Custom Resource option) can’t share the same tls.crt. The tls.crt for external communications should have a Common Name (CN) setting that matches the primary Service name (CN = cluster1-primary.default.svc.cluster.local. in the above example). Similarly, the tls.crt for internal communications should have a Common Name (CN) setting that matches the preset replication user: CN=_crunchyrepl.

One of the options to create certificates yourself is to use CloudFlare PKI and TLS toolkit . Supposing that your cluster name is cluster1 and the desired namespace is postgres-operator, certificates generation may look as follows:

``` {.bash data-prompt="$" }
$ export CLUSTER_NAME=cluster1
$ export NAMESPACE=postgres-operator
$ cat <<EOF | cfssl gencert -initca - | cfssljson -bare ca
{
  "CN": "*",
  "key": {
    "algo": "ecdsa",
    "size": 384
  }
}
EOF

$ cat <<EOF > ca-config.json
{
   "signing": {
     "default": {
        "expiry": "87600h",
        "usages": ["digital signature", "key encipherment", "content commitment"]
      }
   }
}
EOF

$ cat <<EOF | cfssl gencert -ca=ca.pem  -ca-key=ca-key.pem -config=./ca-config.json - | cfssljson -bare server
{
   "hosts": [
     "localhost",
     "${CLUSTER_NAME}-primary",
     "${CLUSTER_NAME}-primary.${NAMESPACE}",
     "${CLUSTER_NAME}-primary.${NAMESPACE}.svc.cluster.local",
     "${CLUSTER_NAME}-primary.${NAMESPACE}.svc"
   ],
   "CN": "${CLUSTER_NAME}-primary.${NAMESPACE}.svc.cluster.local", 
   "key": {
     "algo": "ecdsa",
     "size": 384
   }
}
EOF
```

You can find more on genrating certificates this way in official Kubernetes documentation .

Don’t forget that you should generate certificates twice: one set is for external communications, and another set is for internal ones!

Check your certificates for expiration¶

$ kubectl get secrets

First, check the necessary secrets names (cluster1-cluster-cert and cluster1-replication-cert by default):

You will have the following response:

NAME                            TYPE     DATA   AGE
cluster1-cluster-cert           Opaque   3      11m
...
cluster1-replication-cert       Opaque   3      11m
...

Now use the following command to find out the certificates validity dates, substituting Secrets names if necessary:

$ {
  kubectl get secret/cluster1-replication-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -noout -dates
  kubectl get secret/cluster1-cluster-cert -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -noout -dates
  }

The resulting output will be self-explanatory:

notBefore=Jun 28 10:20:19 2023 GMT
notAfter=Jun 27 11:20:19 2024 GMT
notBefore=Jun 28 10:20:18 2023 GMT
notAfter=Jun 25 11:20:18 2033 GMT

Keep certificates after deleting the cluster¶

In case of cluster deletion, objects, created for SSL (Secret, certificate, and issuer) are not deleted by default.

If the user wants the cleanup of objects created for SSL, there is a finalizers.percona.com/delete-ssl Custom Resource option, which can be set in deploy/cr.yaml: if this finalizer is set, the Operator will delete Secret, certificate and issuer after the cluster deletion event.

Get expert help¶

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services. Join K8S Squad to benefit from early access to features and “ask me anything” sessions with the Experts.

Community Forum Get a Percona Expert Join K8S Squad

Last update: 2024-11-21