Run PMM Client as a Kubernetes pod¶

Run PMM Client as a Kubernetes pod to monitor your databases without installing software on your hosts.

This deployment approach provides:

automatic architecture detection (x86_64/ARM64)
consistent environment across different operating systems
simplified setup and configuration
centralized configuration management via PMM Server

Prerequisites¶

Before deploying PMM Client:

Install kubectl.
Check system requirements to ensure your environment meets the minimum criteria.
Install and configure PMM Server as you’ll need its address to configure the Client. See Configure the PMM Server address.
Set up firewall rules to allow communication between PMM Client and PMM Server.
Create database monitoring users with appropriate permissions for the databases you plan to monitor.
Use PMM Client 3.10.0 or later, otherwise you’ll lose registered Services on every restart regardless of storage. This is because earlier versions re-register the Node on each container start, and PMM Server responds by removing the Node together with every Service on it.

Installation and setup¶

Configure the PMM Server address¶

Find the address of your PMM Server to set as PMM_AGENT_SERVER_ADDRESS when you deploy PMM Client in the next section. Without a working address, the Client can’t register with PMM Server or send it data. How you find it depends on where PMM Server runs:

PMM Server outside the clusterPMM Server in the same cluster

Use its hostname or IP address and port, for example pmm.example.com:443.

Find and qualify the address of PMM Server’s Kubernetes Service.

Find PMM Server’s Service:
```
kubectl get svc -n <namespace>
```
- The standard PMM Server Helm chart names this Service monitoring-service by default.
- For PMM Server in HA mode, register against the HAProxy Service instead, for example pmm-ha-haproxy. The HAProxy Service routes traffic to the active PMM leader; a Service that connects directly to a PMM pod bypasses leader election and does not.
Qualify the Service name with its namespace. A Service name on its own resolves only from inside its own namespace, so a Client deployed to a different namespace needs the full form:
```
- name: PMM_AGENT_SERVER_ADDRESS
  value: monitoring-service.pmm.svc.cluster.local:443
```
An unqualified name fails before the Client even attempts a connection: it can’t look up the address at all, and the pod log names the lookup rather than the connection, which reads like the wrong problem. See the address does not resolve if you hit this.

Deploy PMM Client¶

Choose your deployment approach:

Standalone: Deploy PMM Client as a dedicated pod to monitor external databases
Sidecar: Deploy PMM Client alongside a database in the same pod

Deploy PMM Client as a Standalone containerDeploy PMM Client as a Sidecar container

Deploy PMM Client as a StatefulSet so the pod keeps its name when it restarts. The volume preserves the identity PMM Server issued to the agent, so your monitored services survive restarts and image upgrades.

Follow these steps to deploy PMM Client using kubectl:

(Optional) Create a namespace for the deployment and set it as the default namespace:

kubectl create namespace pmm-client-test
kubectl config set-context --current --namespace=pmm-client-test

Create a Secret to store PMM Server credentials. Replace admin password if you changed it during PMM Server setup:

kubectl create secret generic pmm-secret \
--from-literal=PMM_AGENT_SERVER_USERNAME=admin \
--from-literal=PMM_AGENT_SERVER_PASSWORD=admin

Create pmm-client.yaml to define the StatefulSet and its volume. Replace X.X.X.X:443 with the address of your PMM Server:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pmm-client
spec:
  serviceName: pmm-client
  replicas: 1
  selector:
    matchLabels:
      app: pmm-client
  template:
    metadata:
      labels:
        app: pmm-client
    spec:
      securityContext:
        # The PMM Client image runs as this user, which has to own the volume to write to it.
        fsGroup: 1002
      containers:
        - name: pmm-client
          image: percona/pmm-client:3
          volumeMounts:
            # Holds the Agent identity, so the pod keeps its Node and the Services on it.
            - name: pmm-agent
              mountPath: /usr/local/percona/pmm/config
              subPath: config
            # Holds the metrics buffered on disk while PMM Server is unreachable.
            - name: pmm-agent
              mountPath: /usr/local/percona/pmm/tmp
              subPath: tmp
          env:
            - name: PMM_AGENT_SERVER_ADDRESS
              value: X.X.X.X:443
            - name: PMM_AGENT_SERVER_USERNAME
              valueFrom:
                secretKeyRef:
                  name: pmm-secret
                  key: PMM_AGENT_SERVER_USERNAME
            - name: PMM_AGENT_SERVER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pmm-secret
                  key: PMM_AGENT_SERVER_PASSWORD
            - name: PMM_AGENT_SERVER_INSECURE_TLS
              value: "1"
            # An absolute path, so that it resolves to the mounted volume whatever the
            # container's working directory is.
            - name: PMM_AGENT_CONFIG_FILE
              value: /usr/local/percona/pmm/config/pmm-agent.yaml
            - name: PMM_AGENT_SETUP
              value: "1"
            - name: PMM_AGENT_SETUP_NODE_TYPE
              value: container
            # The pod name of a StatefulSet replica does not change when the pod restarts,
            # which is what lets PMM Server recognise it as the same Node.
            - name: PMM_AGENT_SETUP_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
  volumeClaimTemplates:
    - metadata:
        name: pmm-agent
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 2Gi

Storage

The volume is claimed from your cluster’s default StorageClass. Add storageClassName to the claim template to choose a different one. Keep the size above 1Gi: that is the on-disk queue the Client fills with metrics it cannot deliver while PMM Server is unreachable.

Security and configuration

The PMM_AGENT_SERVER_INSECURE_TLS=1 setting disables TLS certificate verification. For production environments, configure proper TLS certificates and remove this setting.
If disk metrics appear missing or incorrect, your container may not expose /proc/mounts at the default path. Add PMM_AGENT_SETUP_PROC_MOUNTS_PATH to the env section to point PMM Client to the correct location:

- name: PMM_AGENT_SETUP_PROC_MOUNTS_PATH
  value: /path/to/proc/mounts

Deploy PMM Client and configure the pmm-agent in Setup mode to connect to PMM Server:
```
kubectl apply -f pmm-client.yaml
```
Check that the Node registered, and that a restart keeps it:
```
kubectl logs pmm-client-0
kubectl delete pod pmm-client-0
kubectl logs pmm-client-0
```
On the first start the Client registers the Node. After the restart it reports that the Node is registered already and keeps its identity, so any Services you added remain.

Adding services automatically

You can set the container environment variable PMM_AGENT_PRERUN_SCRIPT to a shell script to automatically add services to PMM for monitoring. Because this deployment keeps the Services you add, a prerun script is only needed if you would rather declare them in the manifest than add them once. The script runs on every start of the pod, after setup. pmm-admin add reports a Service which exists already as an error, and a failing script stops the container, so make the script tolerate the Services it added on an earlier start, for example by appending || true to each command.

Recovering a Node after losing the volume

If the volume is deleted while the Node still exists in PMM, the Client cannot register that Node name again and the pod fails to start. Add PMM_AGENT_SETUP_FORCE=1 to the env section for one start to take the name over. PMM Server then removes the old Node together with every Service on it, so remove the variable again afterwards.

Follow these steps to deploy PMM Client as a sidecar alongside MySQL using kubectl:

(Optional) Create a namespace named pmm-client-test for the deployment and set it as the default namespace:

kubectl create namespace pmm-client-test
kubectl config set-context --current --namespace=pmm-client-test

Create mysql-pmm-client-volume.yaml to define persistent storage for MySQL data between pod restarts:

The PMM Client sidecar deliberately gets no volume of its own. It keeps its configuration in the container, registers with PMM_AGENT_SETUP_FORCE=1 and re-adds the database service from PMM_AGENT_PRERUN_SCRIPT on every start, so it rebuilds its state instead of persisting it.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data/mysql"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Create the resources defined in mysql-pmm-client-volume.yaml
```
kubectl apply -f mysql-pmm-client-volume.yaml
```
Create a Secret to store the credentials for PMM Server authentication. Update PMM_AGENT_SERVER_PASSWORD value if you changed the default admin password during setup:
```
kubectl create secret generic pmm-secret \
 --from-literal=PMM_AGENT_SERVER_USERNAME=admin \
 --from-literal=PMM_AGENT_SERVER_PASSWORD=admin
```

Create a Secret to store the MySQL root password:

kubectl create secret generic mysql-secret \
 --from-literal=MYSQL_ROOT_PASSWORD=very_secure_password

Create mysql-pmm-client-pod.yaml to define a Pod running MySQL 9.0 with PMM Client as a sidecar. Replace X.X.X.X:443 with the address of your PMM Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:9
          resources: {}
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: MYSQL_ROOT_PASSWORD
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
        - name: pmm-client
          image: percona/pmm-client:3
          env:
            - name: PMM_AGENT_SERVER_ADDRESS
              value: X.X.X.X:443
            - name: PMM_AGENT_SERVER_USERNAME
              valueFrom:
                secretKeyRef:
                  name: pmm-secret
                  key: PMM_AGENT_SERVER_USERNAME
            - name: PMM_AGENT_SERVER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pmm-secret
                  key: PMM_AGENT_SERVER_PASSWORD
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: MYSQL_ROOT_PASSWORD
            - name: PMM_AGENT_SERVER_INSECURE_TLS
              value: "1"
            - name: PMM_AGENT_CONFIG_FILE
              value: config/pmm-agent.yaml
            - name: PMM_AGENT_SETUP
              value: "1"
            - name: PMM_AGENT_SETUP_FORCE
              value: "1"
            - name: PMM_AGENT_SIDECAR
              value: "1"
            - name: PMM_AGENT_PRERUN_SCRIPT
              value: "pmm-admin status --wait=10s; pmm-admin add mysql --username=root --password=${MYSQL_ROOT_PASSWORD} --query-source=perfschema"
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim

Security note

The PMM_AGENT_SERVER_INSECURE_TLS=1 setting disables TLS certificate verification. For production environments, configure proper TLS certificates and remove this setting.

Deploy MySQL and PMM Client pod:

kubectl apply -f mysql-pmm-client-pod.yaml

View your monitored node¶

To confirm your node is being monitored:

Go to the main menu and select Operating system > Overview.
In the Node Names drop-down menu, select the node you recently registered.
Modify the time range to view the relevant data for your selected node.

Danger

pmm-agent.yaml contains sensitive credentials and should not be shared.

Troubleshooting¶

Failed to register pmm-agent on PMM Server: connection refused¶

If you get Failed to register pmm-agent on PMM Server: connection refused, this typically means that the IP address is incorrect or the PMM Server is unreachable. Verify:

The PMM_AGENT_SERVER_ADDRESS value is correct
PMM Server is running and accessible
Firewall rules allow traffic on port 443

Failed to register pmm-agent on PMM Server: the address does not resolve¶

This error means the pod cannot resolve the name set in PMM_AGENT_SERVER_ADDRESS:

Failed to register pmm-agent on PMM Server: Post "https://monitoring-service:443/v1/management/nodes": dial tcp: lookup monitoring-service on 172.20.0.10:53: server misbehaving.

The message ending varies (no such host is as common as server misbehaving), but lookup is what identifies name resolution as the cause. The usual cause is an unqualified Kubernetes Service name, which resolves only within its own namespace. Qualify the name as <service>.<namespace>.svc.cluster.local as described in Configure the PMM Server address, then verify connectivity:

kubectl exec pmm-client-0 -- curl -sk -o /dev/null -w '%{http_code}\n' \
  https://<service>.<namespace>.svc.cluster.local:443/v1/server/readyz

A 200 response means PMM Server is reachable at that address. The endpoint needs no credentials, so any other response points to an address problem, not the credentials in your Secret.

Pod stuck in Pending state¶

Check that the volume was bound. The standalone StatefulSet claims it through its volume claim template, which names the claim after the template and the pod:

kubectl get pvc
kubectl describe pvc pmm-agent-pmm-client-0

A claim left Pending usually means the cluster has no default StorageClass. List the available classes with kubectl get storageclass and set storageClassName in the claim template.

View PMM Client logs¶

# Standalone deployment
kubectl logs -l app=pmm-client

# Sidecar deployment
kubectl logs -l app=mysql -c pmm-client