Check the Events

Kubernetes Events always provide a wealth of information and should always be checked while troubleshooting issues.

Events can be checked by the following command

$ kubectl get events

Expected output

LAST SEEN   TYPE      REASON                   OBJECT                                                                   MESSAGE
38m         Normal    Provisioning             persistentvolumeclaim/xb-cron-pxc-pxc-backup-stora-202211231300-3qf7g    External provisioner is provisioning volume for claim "default/xb-cron-pxc-pxc-backup-stora-202211231300-3qf7g"
...

Events capture many information happening at Kubernetes level and provide valuable information. By default, the ordering of events cannot be guaranteed. Use the following command to sort the output in a reverse chronological fashion.

$ kubectl get events --sort-by=".lastTimestamp"

Expected output

LAST SEEN   TYPE      REASON                   OBJECT                                                                   MESSAGE
13m         Normal    Created                  pod/xb-cron-pxc-pxc-backup-stora-2022112313300-3qf7g-brxmv               Created container xtrabackup
13m         Normal    Started                  pod/xb-cron-pxc-pxc-backup-stora-2022112313300-3qf7g-brxmv               Started container xtrabackup
12m         Normal    Completed                job/xb-cron-pxc-pxc-backup-stora-2022112313300-3qf7g                     Job completed
...

When there are too many events and there is a need of filtering output, tools like yq , jq can be used to filter specific items or know the structure of the events.

Example:

$ kubectl get events -oyaml | yq .items[0]

Expected output

apiVersion: v1
count: 2
eventTime: null
firstTimestamp: "2022-11-24T05:30:00Z"
involvedObject:
 apiVersion: v1
 kind: Pod
 name: xb-cron-pxc-pxc-backup-stora-202211245300-3qf7g-bpm5s
 namespace: default
 resourceVersion: "41813970"
 uid: c2463e2a-65a0-4fc2-b5c3-86d88bba6b5b
kind: Event
lastTimestamp: "2022-11-24T05:30:03Z"
message: '0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims.'
metadata:
 creationTimestamp: "2022-11-24T05:30:00Z"
 name: xb-cron-pxc-pxc-backup-stora-202211245300-3qf7g-bpm5s.172a6e3851f6710c
 namespace: default
 resourceVersion: "94245"
 uid: d56ea5b8-3b15-4a22-a6ea-a4f641fcc54e
reason: FailedScheduling
reportingComponent: ""
reportingInstance: ""
source:
 component: default-scheduler
type: Warning

Flag --field-selector can be used to filter out the output as well. For example, the following command provides events of Pod only:

$ kubectl get events --field-selector involvedObject.kind=Pod

More fields can be added to the field-selector flag for filtering events further. Example: the following command provides events of Pod by name xb-cron-pxc-pxc-backup-stora-202211245300-3qf7g-bpm5s.

$ kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=xb-cron-pxc-pxc-backup-stora-202211245300-3qf7g-bpm5s

Expected output

LAST SEEN   TYPE      REASON                   OBJECT                                                      MESSAGE
53m         Warning   FailedScheduling         pod/xb-cron-pxc-pxc-backup-stora-202211245300-3qf7g-bpm5s   0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims.
53m         Normal    NotTriggerScaleUp        pod/xb-cron-pxc-pxc-backup-stora-202211245300-3qf7g-bpm5s   pod didn't trigger scale-up: 3 pod has unbound immediate PersistentVolumeClaims

Same way you can query events for other Kubernetes object (StatefulSet, Custom Resource, etc.) to investigate any problems to them:

$ kubectl get events --field-selector involvedObject.kind=PerconaXtraDBCluster,involvedObject.name=cluster1

Expected output

LAST SEEN   TYPE      REASON                     OBJECT                        MESSAGE
10m         Warning   AsyncReplicationNotReady   perconaservermysql/cluster1   cluster1-mysql-1: [not_replicating]
...

Alternatively, you can see events for a specific object in the output of kubectl describe command:

$ kubectl describe ps cluster1

Expected output

Name:         cluster1
...
Events:
  Type     Reason                    Age                From           Message
  ----     ------                    ----               ----           -------
  Warning  AsyncReplicationNotReady  10m (x23 over 13m)    ps-controller  cluster1-mysql-1: [not_replicating]
...

Check kubectl get events --help to know about more options.

Note

It is important to note that events are stored in the etcd for only 60 minutes. Ensure that events are checked within 60 minutes of the issue. Kubernetes cluster administrators might also use event exporters for storing the events.

Get expert help

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services. Join K8S Squad to benefit from early access to features and “ask me anything” sessions with the Experts.

Community Forum Get a Percona Expert Join K8S Squad

---

Last update: 2024-10-29