Initial troubleshooting

Percona Operator for MySQL uses Custom Resources to manage options for the various components of the cluster.

• PerconaXtraDBCluster Custom Resource with Percona XtraDB Cluster options (it has handy pxc shortname also),

• PerconaXtraDBClusterBackup and PerconaXtraDBClusterRestore Custom Resources contain options for Percona XtraBackup used to backup Percona XtraDB Cluster and to restore it from backups (pxc-backup and pxc-restore shortnames are available for them).

The first thing you can check for the Custom Resource is to query it with kubectl get command:

$ kubectl get pxc

Expected output

NAME       ENDPOINT                   STATUS   PXC   PROXYSQL   HAPROXY   AGE
cluster1   cluster1-haproxy.default   ready    3                3         33d

The Custom Resource should have Ready status.

Note

You can check which Percona’s Custom Resources are present and get some information about them as follows:

$ kubectl api-resources | grep -i percona

Expected output

perconaxtradbclusterbackups       pxc-backup,pxc-backups     pxc.percona.com/v1                     true         PerconaXtraDBClusterBackup
perconaxtradbclusterrestores      pxc-restore,pxc-restores   pxc.percona.com/v1                     true         PerconaXtraDBClusterRestore
perconaxtradbclusters             pxc,pxcs                   pxc.percona.com/v1                     true         PerconaXtraDBCluster

Check the Pods

If Custom Resource is not getting Ready status, it makes sense to check individual Pods. You can do it as follows:

$ kubectl get pods

Expected output

NAME                                               READY   STATUS    RESTARTS   AGE
cluster1-haproxy-0                                 2/2     Running   0          6m17s
cluster1-haproxy-1                                 2/2     Running   0          4m59s
cluster1-haproxy-2                                 2/2     Running   0          4m36s
cluster1-pxc-0                                     3/3     Running   0          6m17s
cluster1-pxc-1                                     3/3     Running   0          5m3s
cluster1-pxc-2                                     3/3     Running   0          3m56s
percona-xtradb-cluster-operator-79966668bd-rswbk   1/1     Running   0          9m54s

The above command provides the following insights:

• READY indicates how many containers in the Pod are ready to serve the traffic. In the above example, cluster1-haproxy-0 container has all two containers ready (2/2). For an application to work properly, all containers of the Pod should be ready.
• STATUS indicates the current status of the Pod. The Pod should be in a Running state to confirm that the application is working as expected. You can find out other possible states in the official Kubernetes documentation.
• RESTARTS indicates how many times containers of Pod were restarted. This is impacted by the Container Restart Policy. In an ideal world, the restart count would be zero, meaning no issues from the beginning. If the restart count exceeds zero, it may be reasonable to check why it happens.
• AGE: Indicates how long the Pod is running. Any abnormality in this value needs to be checked.

You can find more details about a specific Pod using the kubectl describe pods <pod-name> command.

$ kubectl describe pods cluster1-pxc-0

Expected output

...
Name:         cluster1-pxc-0
Namespace:    default
...
Controlled By:  StatefulSet/cluster1-pxc
Init Containers:
 pxc-init:
...
Containers:
 pmm-client:
...
 pxc:
...
   Restart Count:  0
   Limits:
     cpu:     1
     memory:  2G
   Requests:
     cpu:      1
     memory:   2G
   Liveness:   exec [/var/lib/mysql/liveness-check.sh] delay=300s timeout=5s period=10s #success=1 #failure=3
   Readiness:  exec [/var/lib/mysql/readiness-check.sh] delay=15s timeout=15s period=30s #success=1 #failure=5
   Environment Variables from:
     pxc-env-vars-pxc  Secret  Optional: true
   Environment:
...
   Mounts:
...
Volumes:
...
Events:                      <none>

This gives a lot of information about containers, resources, container status and also events. So, describe output should be checked to see any abnormalities.

Contact Us

For free technical help, visit the Percona Community Forum.

To report bugs or submit feature requests, open a JIRA ticket.

For paid support and managed or consulting services , contact Percona Sales.

---

Last update: 2023-10-12