Who owns what¶

When something happens in the cluster, the question is which layer reacts: the Operator, Percona XtraDB Cluster (Galera), Kubernetes, or you. This page answers it at a glance. For the detail of what you do and what the Operator does in each area, and for what never to change by hand, see Responsibilities by area. For an overview of the layers, see Design and architecture.

Two rules explain most of the behavior:

You change the cluster only through the Custom Resource and the user Secrets. The Operator turns them into Kubernetes objects, and reverts manual changes to the fields it manages.
Each layer keeps its own part alive. Kubernetes keeps Pods running, Galera keeps the data consistent, and the Operator keeps the objects and the day-2 tasks matching the Custom Resource.

Who reacts when something happens¶

Event	Who reacts	Where it is explained
A container crashes or fails its liveness probe	The kubelet restarts the container	Health checks and probes
A Pod is deleted or evicted	The StatefulSet controller recreates it. The Operator isn’t involved	In-cluster failover
A Kubernetes node fails	Kubernetes. A StatefulSet Pod is recreated elsewhere only after the Node object is removed or the stuck Pod is force-deleted, which is up to you or your platform	Kubernetes
A database node falls behind or rejoins	Galera, with IST or SST	Rejoining the cluster
A proxy loses a database node	The proxy health check. The `pxc-monit` container updates the proxy configuration	Key workflows
The survivor of a 2-of-3 loss is outside the primary component	Liveness restarts it. When no node is in a primary component, the Operator starts the node with the highest `seqno` as a new one	Full cluster crash recovery
You edit the Custom Resource	The Operator reconciles the change	How the Operator reacts to changes
Someone edits an object the Operator manages	The Operator reverts the fields it manages	Change the cluster through the Custom Resource
A scheduled backup is due	The Operator creates the backup object, and a Job runs it	Scheduled backups
A certificate expires and cert-manager isn’t used	You	Update certificates
A disk fills up	You. With storage autoscaling on, the Operator grows the volume	Automatic storage resizing

Change the cluster through the Custom Resource¶

A Custom Resource (CR) is the Kubernetes object that describes the cluster you want. It is a blueprint that the Operator uses to deploy and manage your cluster.

The Custom Resource is the source of truth. Change the cluster by editing the CR.

The Operator compares the CR against what is running about every five seconds and corrects any difference it finds. A change you make directly to a StatefulSet, a Service, a Pod, or a managed Secret is drift. The Operator overwrites it, usually within seconds. If a change survives, this means the Operator does not manage that field yet. This doesn’t mean the change was accepted.

Edits to the fields the Operator manages do not persist. The Operator reverts them and doesn’t raise an event.
Edits outside the CR can still cause damage before they are reverted.

Review responsibilities at a glance¶

The following table shows what the Operator handles and what stays with you. Select an area for the details and the list of manual actions to avoid.

Area	Operator	You	Don’t do manually
Cluster lifecycle	- Creates and heals StatefulSets, Services, and PodDisruptionBudgets. - Rolls out changes with ordered restarts. Kubernetes restarts failed Pods. - Recovers from a full cluster crash.	- Choose the cluster size. - Decide when to pause. - Plan maintenance windows.	- Scale or edit the StatefulSet. - Delete Pods to restart a node. - Remove finalizers.
Backups	- Runs scheduled and on-demand backups. - Enforces deadlines. - Suspends backups on an unhealthy cluster.	- Verify that backups restore. - Set retention. - Keep off-cluster copies.	- Delete a running backup Job. - Run `xtrabackup` by hand against a node.
Restore and point-in-time recovery (PITR)	- Pauses the cluster, restores, and restarts it. - Detects binary log gaps.	- Take a new full backup after a gap. - Pick the restore target.	- Edit the cluster CR during a restore. - Run two restores at once.
TLS	- Issues certificates. - Renews them only via cert-manager. - Applies rotations without downtime.	- Rotate certificates yourself if cert-manager is not installed. - Manage any certificate you supplied.	- Edit the `ssl` or `ssl-internal` Secrets in place. - Delete them to re-issue certificates.
Users and secrets	- Creates the system users with the cluster. - Applies the passwords you change in the Secret to MySQL and the proxies.	- Set passwords in the user Secret. - Manage your own application users.	- Run `ALTER USER` on a system user in MySQL. - Edit the `internal-` Secret. - Drop a system user.
Storage	- Grows Persistent Volume Claims (PVCs) when the CR asks, and automatically when autoscaling is on.	- Pick a StorageClass that supports expansion. - Watch capacity.	- Resize or delete PVCs by hand. - Shrink storage.
Cross-site replication	- Sets up the replication channels from the CR. - Removes channels that are not in the CR. - Logs when a channel isn’t running.	- Provide routable endpoints and matching replication passwords. - Monitor replication and repair it.	- Create replication channels in MySQL by hand.
Upgrades	- Coordinates version upgrades node by node and keeps quorum.	- Decide when to upgrade. - Run major version upgrades as a planned change.	- Bypass SmartUpdate. - Edit image tags to jump a major version.

The Operator¶

The Operator reads the Custom Resource, compares it with what exists in Kubernetes about every five seconds, and creates, updates or deletes objects to close the gap. It also runs the day-2 tasks: ordered restarts (Smart Update), password changes, certificates, storage resizing, backups, restores, point-in-time recovery, cross-site replication channels, and recovery after a full cluster crash.

The Operator does not:

Control Galera at runtime. It writes the Galera configuration, but quorum, SST and IST, and donor selection are handled by Galera itself.
Handle Kubernetes node failures. It does not force-delete Pods stuck on a failed node or react to node taints. Kubernetes recreates a Pod from an unreachable node elsewhere only after the Node object is removed or the stuck Pod is force-deleted.
Validate MySQL options. The contents of spec.pxc.configuration go into a ConfigMap as is.
Install or upgrade CRDs and RBAC.
Drop database users that you remove from spec.users.
Repair broken cross-site replication. It only logs that the replication channel is not running.
Renew self-signed certificates it generated without cert-manager.
Report most problems as Kubernetes events. It emits events only for full cluster crash recovery, CA certificate rotation, and volume resize. Everything else is in the Operator logs and the Custom Resource status.

For each area in detail, see Responsibilities by area. For the objects the Operator creates, see Objects the Operator creates and Services and ports.

Percona XtraDB Cluster and Galera¶

Galera does:

Replicate every transaction to all nodes and certify it, resolving write conflicts.
Maintain quorum and decide which nodes form the primary component that serves queries.
Slow down writes with flow control when a node falls behind.
Bring a joining node up to date: with IST from another node’s write-set cache (gcache) if possible, or with a full SST through Percona XtraBackup otherwise. Galera also chooses the donor node.

What the Operator sets up for Galera:

The list of peers to connect to, from the cluster DNS.
A preferred list of SST donors that excludes cluster1-pxc-0 (the HAProxy writer) and the node with the highest number.
pc.weight=10, unless you set it yourself.
Encryption of replication traffic when internal TLS certificates exist.
innodb_buffer_pool_size and max_connections calculated from the Pod resources, unless you set them yourself. This requires a memory limit (pxc.resources.limits.memory). See Auto-tuning MySQL options.

Health as the Pod probes see it:

A database Pod is ready only when its node belongs to the primary component and is Synced. A node that is a donor counts as ready by default. Set the AVAILABLE_WHEN_DONOR environment variable of the pxc container to a value other than 1 to make a donor not ready. HAProxy’s own health check has a separate switch, OK_IF_DONOR, which is off by default. See In-cluster failover.
The liveness probe tolerates long startups, SST and recovery, so a node that is busy catching up is not killed. A node that stays outside the primary component for about 30 seconds is restarted.
See Health checks and probes for what the probes check and who acts on their result.

What this means in practice:

A Pod that is Running but not Ready usually means Galera hasn’t finished bringing the node in: the node is still receiving an SST or IST, or it cannot reach the other nodes.
An SST that keeps failing stops after pxc.sstRetryCount attempts, and the Pod stays not ready until you intervene. See Limit SST retries.
A bad option in spec.pxc.configuration shows up as mysqld failing to start, in the pxc container logs.

Kubernetes¶

Kubernetes does:

Schedule Pods. The Operator only writes the affinity rules, node selectors, tolerations, priority class and topology spread constraints you specify. The default anti-affinity is required on kubernetes.io/hostname: three database Pods need three schedulable nodes. Otherwise a Pod stays Pending, and after about a minute the Custom Resource status shows the scheduler’s message.
Provide storage. The CSI driver and StorageClass provision volumes, bind them to zones, and expand file systems. A ReadWriteOnce volume ties the Pod to the node or zone where the volume can attach.
Run the network and DNS. The database nodes find each other, and the proxies find the database nodes, by looking up the headless Services in cluster DNS. If DNS is broken, nodes cannot form or join the cluster, proxies lose their backends, and crash detection stops working.
Provision load balancers. For LoadBalancer Services, the cloud provider creates the load balancer. Firewalls and source ranges are yours and the provider’s to manage.
Manage the Pod lifecycle. Run probes, restart failed containers, enforce PodDisruptionBudgets during node drains, and retry failed Jobs.
Collect garbage. Delete objects owned by the Custom Resource when you delete it.

Kubernetes does not:

Know anything about Galera beyond the probes. A native StatefulSet rolling update waits for each Pod to become ready, but it doesn’t pause for a running backup, doesn’t wait for ProxySQL to bring the node back online, and doesn’t stop early when a Pod can’t start.
Expand a volume whose StorageClass doesn’t allow it, or beyond your storage quota.

You, the user¶

The Operator keeps the cluster matching the Custom Resource. Everything it cannot create or decide for you stays with you:

Install and upgrade the CRDs and RBAC, which the Operator never applies. See Upgrade CRD and the Operator.
Provide capacity: enough schedulable nodes for the anti-affinity rules, and a StorageClass that allows volume expansion. See Cluster lifecycle and Storage.
Provide credentials: the passwords in the user Secret, the backup storage credentials, and the PMM token. See Users and Secrets.
Renew certificates when cert-manager isn’t installed, or when you supplied them. See Manage certificates.
Verify that backups restore, set retention, and keep copies outside the cluster storage. See Back up the database.
Monitor cross-site replication, and repair it when it breaks. See Cross-site replication.
Decide when to pause, upgrade, and restore. See Database upgrades and Restore and point-in-time recovery.

Unsafe configurations¶

The Operator refuses configurations that put availability or data at risk, such as fewer than three database nodes or running without TLS. See Stay inside safe operating limits. To use one anyway, you opt in with a flag in the spec.unsafeFlags subsection. See Override a safety check.

Who owns this failure?¶

Find the symptom, check the place named in the second column, and follow the link for the fix.

Cluster and Pods¶

Symptom	Look here first	Usually owned by	What to do
Cluster status is `error` with a message like `wrong PXC options: ...`	`kubectl get pxc cluster1 -o yaml`, the `status` section	User: invalid spec, such as an even size without `unsafeFlags.pxcSize`, both proxies enabled, or Smart Update without a proxy	Fix the Custom Resource. See Custom Resource options
Pod is `Pending`; the cluster status shows a scheduling message	`kubectl describe pod <pod>`, events	Kubernetes and your capacity: required anti-affinity, resource requests, or a volume in another zone	Add nodes or relax the rules. See Control Pod scheduling
Pod is `Running` but not `Ready`	`kubectl logs <pod> -c pxc`	Galera: the node is not `Synced` or not in the primary component, an SST is in progress, or the SST retry limit is reached	See Limit SST retries and Check the logs
`FULL_PXC_CLUSTER_CRASH` in the logs; events `FullClusterCrashDetected` or `AutomaticRecoveryRefused`	Events, the `pxc` container logs	Galera; recovered by the Operator, or by you if automatic recovery is off or refused	See Crash recovery
`mysqld` fails to start after a configuration change	`kubectl logs <pod> -c pxc`	User: `spec.pxc.configuration` is not validated	Fix the option. See Changing MySQL options
Nodes can’t join the cluster; the proxy has no backends	`kubectl logs <proxy-pod> -c pxc-monit`, cluster DNS	Kubernetes networking and DNS	Check that DNS resolves the `cluster1-pxc` Service. See Exec into the containers
A rolling restart is stuck	Operator logs	Operator, waiting for a node to become `Synced`, for a backup to finish, or for a Pod to be scheduled	See Update strategies

Backups and restores¶

Symptom	Look here first	Usually owned by	What to do
A backup has an empty state, or is stuck in `Starting` or `Suspended`	`kubectl get pxc-backup <name> -o yaml`, Operator logs (`Cluster is not ready for backup`, `Another backup is holding the lock`), and with `backup.allowParallel: false`, the `pxc-cluster1-backup-lock` Lease	Cluster health, or, with `allowParallel: false`, another backup holding the lock. An empty state means the Job isn’t created yet	Restore cluster health. See Backup suspension
A backup is `Failed`	`kubectl logs job/xb-<backup>`	User: storage credentials or endpoint. Galera: the donor node left the cluster	See Configure storage for backups
Backup condition `PITRReady=False` with reason `BinlogGapDetected`	Backup conditions, `kubectl logs deployment/cluster1-pitr`	User: binlog retention on the servers is too short	Take a new full backup. See Binlog gaps
A restore is `Failed`	`kubectl get pxc-restore <name> -o yaml` (`status.comments`), logs of the `restore-job-`, `prepare-job-` and `pitr-job-*` Jobs	Varies: often the backup source or storage access	See Restore
A scheduled backup didn’t run	Operator logs	The cluster wasn’t `ready` when the run was due, so the run was skipped; or the Operator wasn’t running at that time	See Scheduled backups
A backup object is stuck in `Terminating`	Operator logs	The backup files could not be deleted from storage, so the `percona.com/delete-backup` finalizer is not removed	Check storage access. See Delete the unneeded backup

Storage, replication and TLS¶

Symptom	Look here first	Usually owned by	What to do
A storage size change does nothing	The `storage` value in the Custom Resource compared with the PVC size; events	User, if `storageScaling.enableVolumeScaling` is off. Kubernetes or CSI, for `StorageClassNotSupportResize`, `ExceededQuota` or `VolumeResizeFailed` events	See Resize storage and Troubleshoot storage resizing
A cross-site replica is lagging or broken	Operator logs (`Replication for channel is not running`), `SHOW REPLICA STATUS` on the replica	MySQL replication; you fix it	See Set up cross-site replication
TLS errors after about 90 days, without cert-manager	Certificate expiry date in the `cluster1-ssl` Secret	User: certificates need manual renewal	See Update certificates

How to tell which layer it is¶

Check these sources in order. Each one narrows down the layer.

Custom Resource status and conditions. kubectl get pxc cluster1 -o yaml shows the cluster state, the reconcile error message if any, and scheduling problems. An error state with a spec message points to the user. See Status.
Events. kubectl get events --sort-by=.lastTimestamp shows Kubernetes problems: scheduling, volume attachment and resize, image pulls, probe failures. The Operator itself adds events only for full cluster crash recovery, CA rotation and volume resize. See Check the events.
Operator logs. kubectl logs deployment/percona-xtradb-cluster-operator shows what the Operator is doing or waiting for: a stuck rolling restart, a skipped backup, broken replication. Most Operator problems appear only here.
Pod and container logs. kubectl logs cluster1-pxc-0 -c pxc shows Galera and MySQL problems: SST failures, quorum loss, mysqld start failures. For backup and restore problems, read the logs of the Job. See Check the logs.

Next steps¶

Responsibilities by area