If PMM server wasn’t updated properly, or if you have concerns about the release, you can force the update process in 2 ways:
From the UI - Home panel: click with the Alt key on the reload icon in the Update panel to make the Update Button visible even if you are on the same version as available for update. Pressing this button will force the system to rerun the update so that any broken or not installed components can be installed. In this case, you’ll go through the usual update process with update logs and successful messages at the end.
By API call (if UI not available): You can call the Update API directly with:
curl --user admin:admin --request POST 'http://PMM_SERVER/v1/Updates/Start'
admin:adminwith your username/password, and replace
PMM_SERVERwith your server address.
You will not see the logs using this method.
Refresh The Home page in 2-5 minutes and you should see that PMM was updated.
There are many causes of broken network connectivity.
When using Docker the container is constrained by the host-level routing and firewall rules. For example, your hosting provider might have default
iptables rules on their hosts that block communication between PMM Server and PMM Client, resulting in DOWN targets in VictoriaMetrics. If this happens, check the firewall and routing settings on the Docker host.
PMM is also able to generate diagnostics data which can be examined and/or shared with our support team to help solve an issue. You can get collected logs from PMM Client using the
pmm-admin summary command.
Logs obtained in this way includes PMM Client logs and logs which were received from the PMM Server, stored separately in the
server folders. The
server folder also contains its own
client subfolder with the self-monitoring client information collected on the PMM Server.
You can get PMM Server logs with either of these methods:
In a browser, visit
From Help menu¶
Select Help → PMM Logs.
Click PMM Logs to retrieve PMM diagnostics data which can be examined and shared with our support team should you need help.
When adding a service, the host might not be detected if the password contains special symbols (e.g.
In such cases, you should convert any password, replacing special characters with their escape sequence equivalents.
One way to do this is to use the
When adding clients to the PMM server, you use the
admin user. However, if you change the password for the admin user from the PMM UI, then the clients will not be able to access PMM due to authentication issues. Also, due to multiple unsuccessful login attempts, Grafana will lock out the admin user.
In such a scenario, use API key for authentication. You can use API keys as a replacement for basic authentication.
No Integrated Alerting icon¶
You are not logged in as a privileged user. You need either Admin or Editor roles to work with Integrated Alerting.
Integrated Alerting icon but no submenu¶
Integrated Alerting isn’t activated.
- Go to Configuration → Settings → Advanced Settings.
- Enable Integrated Alerting.
Unreachable external IP addresses¶
When I get an email or page from my system the IP is not reachable from outside my organization how do I fix this?
To configure your PMM Server’s Public Address, select Configuration → Settings → Advanced Settings, and supply an address to use in your alert notifications.
What is ‘Alertmanager integration’?¶
There’s already an Alertmanager integration tab without me turning it on, I know because I was using your existing Alertmanager integration.
This will continue to work but will be renamed External Alertmanager.
Notification channels not working¶
I tried to setup a Slack/Email channel but nothing happened.
Before you can use a notification channel you must provide your connection details.
- Go to PMM → PMM Settings → Communication.
- Define your SMTP server or Slack incoming webhook URL.
For PagerDuty you can configure in the notification channel tab of Integrated Alerting by supplying your server/routing key.
What’s the difference: Username/Password vs Identity/Secret¶
In configuring my email server I’m being asked for a Username and Password as well as Identity and Secret. What is the difference between these and which do I use or do I need both?
It depends on what kind of authentication your system uses:
LOGIN: Use Username/Password.
PLAIN: Use either Username or Identity and Password.
CRAM-MD5: Use Username and Secret.
Alert Rule Templates is disabled¶
Built-In alerts are not editable, but you can copy them and edit the copies. (In PMM 2.14.0 and above).
If you create a custom alert rule template you will have access to edit.
I’m ready to create my first rule! I’ve chosen a template and given it a name…what is the format of the fields?
Threshold - float value, it has different meanings depending on what template is used.
Duration - The duration the condition must be satisfied in seconds.
Filters - A Key, Evaluator, and Value. E.g.
Key must be an exact match. You can find a complete list of keys by using the Explore main menu item in PMM.
Evaluator can be:
Value is an exact match or when used with a ‘fuzzy’ evaluator (
=~) can be a regular expression. E.g.
Variables in Templates¶
The concept of template implies features like variable substitutions…where can I use these? Where can I find a complete list of them?
Here is a guide to creating templates for Alertmanager: https://prometheus.io/docs/prometheus/latest/configuration/template_examples/
Why don’t I see the whole query?
To reduce space usage, long query examples and fingerprints can be truncated to 1024 symbols. In this case, the query explains section will not work.