Initial troubleshooting¶
Percona Operator for MySQL uses Custom Resources to manage options for the various components of the cluster.
-
PerconaXtraDBCluster
Custom Resource with Percona XtraDB Cluster options (it has handypxc
shortname also), -
PerconaXtraDBClusterBackup
andPerconaXtraDBClusterRestore
Custom Resources contain options for Percona XtraBackup used to backup Percona XtraDB Cluster and to restore it from backups (pxc-backup
andpxc-restore
shortnames are available for them).
The first thing you can check for the Custom Resource is to query it with kubectl get
command:
$ kubectl get pxc
Expected output
NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE
cluster1 cluster1-haproxy.default ready 3 3 33d
The Custom Resource should have Ready
status.
Note
You can check which Percona’s Custom Resources are present and get some information about them as follows:
$ kubectl api-resources | grep -i percona
Expected output
perconaxtradbclusterbackups pxc-backup,pxc-backups pxc.percona.com/v1 true PerconaXtraDBClusterBackup
perconaxtradbclusterrestores pxc-restore,pxc-restores pxc.percona.com/v1 true PerconaXtraDBClusterRestore
perconaxtradbclusters pxc,pxcs pxc.percona.com/v1 true PerconaXtraDBCluster
Check the Pods¶
If Custom Resource is not getting Ready
status, it makes sense to check
individual Pods. You can do it as follows:
$ kubectl get pods
Expected output
NAME READY STATUS RESTARTS AGE
cluster1-haproxy-0 2/2 Running 0 6m17s
cluster1-haproxy-1 2/2 Running 0 4m59s
cluster1-haproxy-2 2/2 Running 0 4m36s
cluster1-pxc-0 3/3 Running 0 6m17s
cluster1-pxc-1 3/3 Running 0 5m3s
cluster1-pxc-2 3/3 Running 0 3m56s
percona-xtradb-cluster-operator-79966668bd-rswbk 1/1 Running 0 9m54s
The above command provides the following insights:
READY
indicates how many containers in the Pod are ready to serve the traffic. In the above example,cluster1-haproxy-0
Pod has all two containers ready (2/2). For an application to work properly, all containers of the Pod should be ready.STATUS
indicates the current status of the Pod. The Pod should be in aRunning
state to confirm that the application is working as expected. You can find out other possible states in the official Kubernetes documentation .RESTARTS
indicates how many times containers of Pod were restarted. This is impacted by the Container Restart Policy . In an ideal world, the restart count would be zero, meaning no issues from the beginning. If the restart count exceeds zero, it may be reasonable to check why it happens.AGE
: Indicates how long the Pod is running. Any abnormality in this value needs to be checked.
You can find more details about a specific Pod using the
kubectl describe pods <pod-name>
command.
$ kubectl describe pods cluster1-pxc-0
Expected output
...
Name: cluster1-pxc-0
Namespace: default
...
Controlled By: StatefulSet/cluster1-pxc
Init Containers:
pxc-init:
...
Containers:
pmm-client:
...
pxc:
...
Restart Count: 0
Limits:
cpu: 1
memory: 2G
Requests:
cpu: 1
memory: 2G
Liveness: exec [/var/lib/mysql/liveness-check.sh] delay=300s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [/var/lib/mysql/readiness-check.sh] delay=15s timeout=15s period=30s #success=1 #failure=5
Environment Variables from:
pxc-env-vars-pxc Secret Optional: true
Environment:
...
Mounts:
...
Volumes:
...
Events: <none>
This gives a lot of information about containers, resources, container status and also events. So, describe output should be checked to see any abnormalities.