Check the Events

Kubernetes Events always provide a wealth of information and should always be checked while troubleshooting issues.

Events can be checked by the following command

$ kubectl get events

Expected output

LAST SEEN   TYPE      REASON                   OBJECT                                                MESSAGE
3s          Normal    Provisioning             persistentvolumeclaim/datadir-cluster1-mysql-2        External provisioner is provisioning volume for claim "default/datadir-cluster1-mysql-2"
3s          Normal    ProvisioningSucceeded    persistentvolumeclaim/datadir-cluster1-mysql-2        Successfully provisioned volume pvc-fbd347c2-adf7-413b-86bb-b5e381313cc0
...

Events capture many information happening at Kubernetes level and provide valuable information. By default, the ordering of events cannot be guaranteed. Use the following command to sort the output in a reverse chronological fashion.

$ kubectl get events --sort-by=".lastTimestamp"

Expected output

LAST SEEN   TYPE      REASON                   OBJECT                                                MESSAGE
33m         Warning   FailedScheduling           pod/cluster1-mysql-0                                  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
33m         Normal    Provisioning               persistentvolumeclaim/datadir-cluster1-mysql-0        External provisioner is provisioning volume for claim "default/datadir-cluster1-mysql-0"
33m         Normal    ProvisioningSucceeded      persistentvolumeclaim/datadir-cluster1-mysql-0        Successfully provisioned volume pvc-aad3d7cf-2bd4-4823-8e6f-38b9a8528aaa
...

When there are too many events and there is a need of filtering output, tools like yq , jq can be used to filter specific items or know the structure of the events.

Example:

$ kubectl get events -oyaml | yq .items[11]

Expected output

apiVersion: v1
count: 1
eventTime: null
firstTimestamp: "2024-07-16T08:57:32Z"
involvedObject:
  apiVersion: v1
  kind: Pod
  name: cluster1-mysql-0
  namespace: default
  resourceVersion: "623"
  uid: 689338c7-d5f7-4bfb-9f7e-ca1d13e782a3
kind: Event
lastTimestamp: "2024-07-16T08:57:32Z"
message: '0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.'
metadata:
  creationTimestamp: "2024-07-16T08:57:32Z"
  name: cluster1-mysql-0.17e2a5c04f66588f
  namespace: default
  resourceVersion: "625"
  uid: 6d4a0eeb-8eea-4e1d-91f5-91fb795bd743
reason: FailedScheduling
reportingComponent: default-scheduler
reportingInstance: ""
source:
  component: default-scheduler
type: Warning

Flag --field-selector can be used to filter out the output as well. For example, the following command provides events of Pod only:

$ kubectl get events --field-selector involvedObject.kind=Pod

More fields can be added to the field-selector flag for filtering events further. For example, the following command provides events of the cluster1-mysql-0 Pod:

$ kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=cluster1-mysql-0

Expected output

LAST SEEN   TYPE      REASON            OBJECT                 MESSAGE
53m         Normal    Created           pod/cluster1-mysql-0   Created container mysql
53m         Normal    Started           pod/cluster1-mysql-0   Started container mysql
53m         Normal    Pulling           pod/cluster1-mysql-0   Pulling image "percona/percona-server-mysql-operator:0.8.0-backup"
...

Same way you can query events for other Kubernetes object (StatefulSet, Custom Resource, etc.) to investigate any problems to them:

$ kubectl get events --field-selector involvedObject.kind=PerconaServerMySQL,involvedObject.name=cluster1

Expected output

LAST SEEN   TYPE      REASON                     OBJECT                        MESSAGE
10m         Warning   AsyncReplicationNotReady   perconaservermysql/cluster1   cluster1-mysql-1: [not_replicating]
...

Alternatively, you can see events for a specific object in the output of kubectl describe command:

$ kubectl describe ps cluster1

Expected output

Name:         cluster1
...
Events:
  Type     Reason                    Age                From           Message
  ----     ------                    ----               ----           -------
  Warning  AsyncReplicationNotReady  10m (x23 over 13m)    ps-controller  cluster1-mysql-1: [not_replicating]
...

Check kubectl get events --help to know about more options.

Note

It is important to note that events are stored in the etcd for only 60 minutes. Ensure that events are checked within 60 minutes of the issue. Kubernetes cluster administrators might also use event exporters for storing the events.

Get expert help

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services. Join K8S Squad to benefit from early access to features and “ask me anything” sessions with the Experts.

Community Forum Get a Percona Expert Join K8S Squad

---

Last update: 2024-10-29