Skip to content
logo
Percona Operator for MySQL
Debug and troubleshoot
Initializing search
    percona/k8spxc-docs
    percona/k8spxc-docs
    • Welcome
      • System Requirements
      • Design and architecture
      • Comparison with other solutions
      • Install with Helm
      • Install with kubectl
      • Install on Minikube
      • Install on Google Kubernetes Engine (GKE)
      • Install on Amazon Elastic Kubernetes Service (AWS EKS)
      • Install on Microsoft Azure Kubernetes Service (AKS)
      • Install on OpenShift
      • Generic Kubernetes installation
      • Multi-cluster and multi-region deployment
      • Application and system users
      • Changing MySQL Options
      • Anti-affinity and tolerations
      • Labels and annotations
      • Local Storage support
      • Defining environment variables
      • Load Balancing with HAProxy
      • Load Balancing with ProxySQL
      • Transport Encryption (TLS/SSL)
      • Data at rest encryption
      • Telemetry
      • Backup and restore
      • Upgrade Database and Operator
      • Horizontal and vertical scaling
      • Monitor with Percona Monitoring and Management (PMM)
      • Add sidecar containers
      • Restart or pause the cluster
      • Crash recovery
      • Debug and troubleshoot
        • Cluster-level logging
        • Avoid the restart-on-fail loop for Percona XtraDB Cluster containers
        • Special debug images
      • How to install Percona XtraDB Cluster in multi-namespace (cluster-wide) mode
      • How to upgrade Percona XtraDB Cluster manually
      • How to use private registry
      • Custom Resource options
      • Percona certified images
      • Operator API
      • Frequently Asked Questions
      • Old releases (documentation archive)
      • Release notes index
      • Percona Operator for MySQL based on Percona XtraDB Cluster 1.12.0 (2022-12-07)
      • Percona Operator for MySQL based on Percona XtraDB Cluster 1.11.0 (2022-06-03)
      • Percona Distribution for MySQL Operator 1.10.0 (2021-11-24)
      • Percona Distribution for MySQL Operator 1.9.0 (2021-08-09)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.8.0 (2021-05-26)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.7.0 (2021-02-02)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.6.0 (2020-09-09)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.5.0 (2020-07-21)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.4.0 (2020-04-29)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.3.0 (2020-01-06)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.2.0 (2019-09-20)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.1.0 (2019-07-15)
      • Percona Kubernetes Operator for Percona XtraDB Cluster 1.0.0 (2019-05-29)

    • Cluster-level logging
    • Avoid the restart-on-fail loop for Percona XtraDB Cluster containers
    • Special debug images

    Debug¶

    Cluster-level logging¶

    Cluster-level logging involves collecting logs from all Percona XtraDB Cluster Pods in the cluster to some persistent storage. This feature gives the logs a lifecycle independent of nodes, Pods and containers in which they were collected. Particularly, it ensures that Pod logs from previous failures are available for later review.

    Log collector is turned on by the logcollector.enabled key in the deploy/cr.yaml configuration file (true by default).

    The Operator collects logs using Fluent Bit Log Processor, which supports many output plugins and has broad forwarding capabilities. If necessary, Fluent Bit filtering and advanced features can be configured via the logcollector.configuration key in the deploy/cr.yaml configuration file.

    Logs are stored for 7 days and then rotated.

    Collected logs can be examined using the following command:

    $ kubectl logs cluster1-pxc-1 -c logs
    

    Note

    Technically, logs are stored on the same Persistent Volume, which is used with the corresponding Percona XtraDB Cluster Pod. Therefore collected logs can be found in DATADIR (var/lib/mysql/).

    Note

    You can parse output of the logs with jq JSON processor as follows: kubectl logs cluster1-pxc-1 -c logs -f | jq -R 'fromjson?'.

    Avoid the restart-on-fail loop for Percona XtraDB Cluster containers¶

    The restart-on-fail loop takes place when the container entry point fails (e.g. mysqld crashes). In such a situation, Pod is continuously restarting. Continuous restarts prevent to get console access to the container, and so a special approach is needed to make fixes.

    You can prevent such infinite boot loop by putting the Percona XtraDB Cluster containers into the infinity loop without starting mysqld. This behavior of the container entry point is triggered by the presence of the /var/lib/mysql/sleep-forever file.

    For example, you can do it for the pxc container of an appropriate Percona XtraDB Cluster instance as follows:

    $ kubectl exec -it cluster1-pxc-0 -c pxc -- sh -c 'touch /var/lib/mysql/sleep-forever'
    

    If pxc container can’t start, you can use logs container instead:

    $ kubectl exec -it cluster1-pxc-0 -c logs -- sh -c 'touch /var/lib/mysql/sleep-forever'
    

    The instance will restart automatically and run in its usual way as soon as you remove this file (you can do it with a command similar to the one you have used to create the file, just substitute touch to rm in it).

    Special debug images¶

    For the cases when Pods are failing for some reason or just show abnormal behavior, the Operator can be used with a special debug images. Percona XtraDB Cluster debug image has the following specifics:

    • it avoids restarting on fail,

    • it contains additional tools useful for debugging (sudo, telnet, gdb, etc.),

    • it has debug mode enabled for the logs.

    There are debug versions for all Percona XtraDB Cluster images: they have same names as normal images with a special -debug suffix in their version tag: for example, percona-xtradb-cluster:8.0.29-21.1-debug.

    To use the debug image instead of the normal one, find the needed image name in the list of certified images and set it for the proper key in the deploy/cr.yaml configuration file. For example, set the following value of the pxc.image key to use the Percona XtraDB Cluster debug image:

    • percona/percona-xtradb-cluster:8.0.29-21.1-debug for Percona XtraDB Cluster 8.0,

    • percona/percona-xtradb-cluster:5.7.39-31.61-debug for Percona XtraDB Cluster 5.7.

    The Pod should be restarted to get the new image.

    Note

    When the Pod is continuously restarting, you may have to delete it to apply image changes.

    Contact Us

    For free technical help, visit the Percona Community Forum.

    To report bugs or submit feature requests, open a JIRA ticket.

    For paid support and managed or consulting services , contact Percona Sales.


    Last update: 2023-02-09
    Back to top
    Percona LLC and/or its affiliates, © 2009 - 2022
    Made with Material for MkDocs

    Cookie consent

    We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better.