Skip to content
logo
Percona Monitoring and Management
Resolve issues
Initializing search
    percona/pmm-doc
    percona/pmm-doc
    • Welcome
    • Setting up
      • Server
        • Network
        • Docker
        • Podman
        • Helm
        • Virtual Appliance
        • AWS Marketplace
        • Easy-install script
      • Client
        • MySQL
        • MongoDB
        • PostgreSQL
        • ProxySQL
        • Amazon RDS
        • Microsoft Azure
        • Google Cloud Platform
        • Linux
        • External Services
        • HAProxy
        • Remote instances
    • Get started
      • User Interface
      • Percona Alerting
      • Backup and Restore
        • Prepare a storage location
        • MongoDB backups
          • MongoDB backup prerequisites
          • Create MongoDB on-demand and scheduled backups
          • Create MongoDB PITR backups
          • Restore a MongoDB backup
          • MongoDB Backup and Restore support matrix
        • MySQL backups
          • MySQL backup prerequisites
          • Create a MySQL backup
          • Restore a MySQL backup
        • Edit a scheduled backup
        • Delete a backup
      • Roles and permissions
        • Configure access control
        • Labels for access control
        • Create access roles
        • Manage access roles
        • Assign roles to users
        • Use Case
      • Query Analytics
      • Working with Advisors
    • How to
      • Configure
      • Manage users
      • Upgrade
      • Secure
      • Optimize
      • Annotate
      • Share dashboards and panels
      • Extend Metrics
      • Resolve issues
        • Troubleshooting checklist
        • Troubleshooting areas
          • Upgrade issues
            • PMM server not updating correctly
            • PMM server not showing latest versions available with the instances created from AWS
            • PMM server fails while upgrading
          • Configuration issues
            • Client-server connections
            • Connection difficulties
        • Percona Alerting
          • No Alert rule templates tab on the Alerting page
          • Custom alert rule templates not migrated to Percona Alerting
            • Unreachable external IP addresses
            • Alert Rule Templates are disabled
          • QAN issues
            • Missing data
          • Plugins issues
      • Integrate with Percona Platform
        • Check Percona Portal account information
    • Details
      • Architecture
      • UI components
      • PMM components and versions
      • Data handling in PMM
      • Develop Advisor checks
      • PMM Inventory
      • Dashboards
        • Manage dashboards
          • Insight
            • Advanced Data Exploration
            • VictoriaMetrics
            • VictoriaMetrics Agents Overview
          • Environments
            • Environment Summary
          • Kubernetes
            • Kubernetes Pods Status
            • Kubernetes Volumes
          • DBaas
          • OS Dashboards
            • Disk Details
            • Network Details
            • Memory Details
            • Node Temperature Details
            • Nodes Compare
            • Nodes Overview
            • Node Summary
            • NUMA Details
            • Processes Details
          • Prometheus Dashboards
            • Prometheus Exporters Overview
          • MySQL Dashboards
            • MySQL Command/Handler Counters Compare
            • MySQL InnoDB Compression Details
            • MySQL InnoDB Details
            • MySQL MyISAM/Aria Details
            • MySQL MyRocks Details
            • MySQL Instance Summary
            • MySQL Instances Compare
            • MySQL Instances Overview
            • MySQL Wait Event Analyses Details
            • MySQL Performance Schema Details
            • MySQL Query Response Time Details
            • MySQL Replication Summary
            • MySQL Group Replication Summary
            • MySQL Table Details
            • MySQL User Details
            • MySQL TokuDB Details
          • MongoDB Dashboards
            • Experimental MongoDB Collection Details
            • Experimental MongoDB Oplog Details
            • MongoDB Cluster Summary
            • MongoDB Instance Summary
            • MongoDB Instances Overview
            • MongoDB Instances Compare
            • MongoDB ReplSet Summary
            • MongoDB InMemory Details
            • MongoDB MMAPv1 Details
            • MongoDB WiredTiger Details
          • PostgreSQL Dashboards
            • PostgreSQL Instance Summary
            • PostgreSQL Instances Compare
            • Experimental PostgreSQL Vacuum Monitoring
          • ProxySQL Dashboards
          • HA Dashboards
            • PXC/Galera Cluster Summary
            • Experimental PXC/Galera Cluster Summary
            • PXC/Galera Nodes Compare
            • HAProxy Instance Summary
      • Commands
        • pmm-admin - PMM Administration Tool
        • pmm-agent - PMM Client agent
      • API
      • VictoriaMetrics
      • ClickHouse
      • PostgreSQL
      • Glossary
      • Introduction
      • DBaaS architecture
        • Setting up DBaaS
        • Create a Kubernetes Cluster
        • Deleting Kubernetes clusters
        • Activating DBaaS
        • Add a Kubernetes cluster automatically
        • Add a Kubernetes cluster manually
        • Manage allowed component versions
          • OLM installation
          • Operators installation
        • Add a DB Cluster
        • Manage a DB Cluster
        • Delete a DB Cluster
        • Create a database cluster from a template
      • Backup and restore
    • FAQ
    • Release Notes
      • PMM 2.39.0
      • PMM 2.38.1
      • PMM 2.38.0
      • PMM 2.37.1
      • PMM 2.37.0
      • PMM 2.36.0
      • PMM 2.35.0
      • PMM 2.34.0
      • PMM 2.33.0
      • PMM 2.32.0
      • PMM 2.31.0
      • PMM 2.30.0
      • PMM 2.29.1
      • PMM 2.29.0
      • PMM 2.28.0
      • PMM 2.27.0
      • PMM 2.26.0
      • PMM 2.25.0
      • PMM 2.24.0
      • PMM 2.23.0
      • PMM 2.22.0
      • PMM 2.21.0
      • PMM 2.20.0
      • PMM 2.19.0
      • PMM 2.18.0
      • PMM 2.17.0
      • PMM 2.16.0
      • PMM 2.15.1
      • PMM 2.15.0
      • PMM 2.14.0
      • PMM 2.13.0
      • PMM 2.12.0
      • PMM 2.11.1
      • PMM 2.11.0
      • PMM 2.10.1
      • PMM 2.10.0
      • PMM 2.9.1
      • PMM 2.9.0
      • PMM 2.8.0
      • PMM 2.7.0
      • PMM 2.6.1
      • PMM 2.6.0
      • PMM 2.5.0
      • PMM 2.4.0
      • PMM 2.3.0
      • PMM 2.2.2
      • PMM 2.2.1
      • PMM 2.2.0
      • PMM 2.1.0
      • PMM 2.0.1
      • PMM 2.0.0
      • Copyright and licensing information
      • Trademark policy

    • Troubleshooting checklist
    • Troubleshooting areas
      • Upgrade issues
        • PMM server not updating correctly
        • PMM server not showing latest versions available with the instances created from AWS
        • PMM server fails while upgrading
      • Configuration issues
        • Client-server connections
        • Connection difficulties
    • Percona Alerting
      • No Alert rule templates tab on the Alerting page
      • Custom alert rule templates not migrated to Percona Alerting
        • Unreachable external IP addresses
        • Alert Rule Templates are disabled
      • QAN issues
        • Missing data
      • Plugins issues

    Resolve issues¶

    This section describes solutions to common problems and scenarios you might encounter while using PMM.

    Troubleshooting checklist¶

    The following questions might help you identify the origin of the problem while using Percona Monitoring and Management:

    1. Are you using the latest PMM version?
    2. Did you check the known issues section in the Release Notes for that particular PMM release?
    3. Are you receiving any error messages?
    4. Do the logs contain any messages about the problem? See Message logs and Trace logs for more information.
    5. Does the problem occur while configuring PMM, such as:
      • Does the problem occur while you configure a specific function?
      • Does the problem occur when you perform a particular task?
    6. Are you using the recommended authentication method?
    7. Does your system’s firewall allow TCP traffic on the ports used by PMM?
    8. Have you allocated enough disk space for installing PMM? If not, check the disk allocation space.
    9. Are you using a Technical Preview feature? Technical Preview features are not production-ready and should only be used in testing environments. For more information, see the relevant Release Notes.
    10. For installing the PMM client, are you using a package other than a binary package without root permissions?
    11. Is your PMM Server installed and running with a known IP address accessible from the client node?
    12. Is the PMM Client installed, and is the node registered with PMM Server?
    13. Is PMM-client configured correctly and has access to the config file?
    14. For monitoring MongoDB, do you have adminUserAnyDatabase or superuser role privilege to any database servers you want to monitor?
    15. For monitoring Amazon RDS using PMM, is there too much latency between PMM Server and the Amazon RDS instance?
    16. Have you upgraded the PMM Server before you upgraded the PMM Client? If yes, there might be configuration issues, thus leading to failure in the client-server communication, as PMM Server might not be able to identify all the parameters in the configuration.
    17. Is the PMM Server version higher than or equal to the PMM Client version? Otherwise, there might be configuration issues, thus leading to failure in the client-server communication, as PMM Server might not be able to identify all the parameters in the configuration.

    Troubleshooting areas¶

    Upgrade issues¶

    PMM server not updating correctly¶

    If the PMM server wasn’t updated correctly, or if you have concerns about the release, you can force the update process in 2 ways:

    1. From the UI - Home panel: click the Alt key on the reload icon in the Update panel to make the Update Button visible even if you are on the same version as available for update. Pressing this button will force the system to rerun the update so that any broken or not installed components can be installed. In this case, you’ll go through the usual update process with update logs and successful messages at the end.

    2. By API call (if UI not available): You can call the Update API directly with:

      curl --user admin:admin --request POST 'http://PMM_SERVER/v1/Updates/Start'
      

      Replace admin:admin with your username/password, and replace PMM_SERVER with your server address.

      You will not see the logs using this method.

      Refresh The Home page in 2-5 minutes, and you should see that PMM was updated.

    3. Upgrade PMM server using Docker.

    PMM server not showing latest versions available with the instances created from AWS¶

    For PMM versions prior to 2.33.0, in specific environments, including AWS, some EPEL repository mirrors did not respond within the time limit defined by pmm-update (currently set to 30 seconds). It was causing supervisord to kill pmm-update-checker, which determines if a newer PMM Server is available for upgrade.

    Solution

    Log in to the PMM Server and run the following command as a root user:

       $ yum-config-manager --setopt=epel.timeout=1 --save
    

    PMM server fails while upgrading¶

    A bug in PMM Server ansible scripts caused PMM to upgrade Nginx’s dependencies without updating Nginx itself. Due to this, PMM throws an error while upgrading and cannot upgrade to a newer version.

    Important

    This issue has been resolved for PMM version 2.33.0. However, the issue persists on all the versions prior to 2.33.0.

    Solution

    While PMM is being upgraded, log in to the PMM server and run the following command:

       sed -i 's/- nginx/- nginx*/' /usr/share/pmm-update/ansible/playbook/tasks/update.yml
    

    Configuration issues¶

    This section focuses on configuration issues, such as PMM-agent connection, adding and removing services for monitoring, and so on.

    Client-server connections¶

    There are many causes of broken network connectivity.

    The container is constrained by the host-level routing and firewall rules when using using Docker. For example, your hosting provider might have default iptables rules on their hosts that block communication between PMM Server and PMM Client, resulting in DOWN targets in VictoriaMetrics. If this happens, check the firewall and routing settings on the Docker host.

    PMM can also generate diagnostics data that can be examined and/or shared with our support team to help solve an issue. You can get collected logs from PMM Client using the pmm-admin summary command.

    Logs obtained in this way include PMM Client logs and logs received from the PMM Server, and stored separately in the client and server folders. The server folder also contains its client subfolder with the self-monitoring client information collected on the PMM Server.

    Beginning with PMM 2.4.0, there is a flag that enables the fetching of pprof debug profiles and adds them to the diagnostics data. To enable, run pmm-admin summary --pprof.

    You can get PMM Server logs with either of these methods:

    Direct download

    In a browser, visit https://<address-of-your-pmm-server>/logs.zip.

    From Help menu

    1. Select Help → PMM Logs.

    2. Click PMM Logs to retrieve PMM diagnostics data which can be examined and shared with our support team should you need help.

    Connection difficulties¶

    Passwords

    When adding a service, the host might not be detected if the password contains special symbols (e.g., @, %, etc.).

    In such cases, you should convert any password, replacing special characters with their escape sequence equivalents.

    One way to do this is to use the encodeURIComponent JavaScript function in your browser’s web console (commonly found under a Development Tools menu). Run the function with your password as the parameter. For example:

    > encodeURIComponent("s3cR#tpa$$worD")
    

    will give:

    "s3cR%23tpa%24%24worD"
    

    Password change

    When adding clients to the PMM server, you use the admin user. However, if you change the password for the admin user from the PMM UI, then the clients will not be able to access PMM due to authentication issues. Also, Grafana will lock out the admin user due to multiple unsuccessful login attempts.

    In such a scenario, use API key for authentication. You can use API keys as a replacement for basic authentication.

    Percona Alerting¶

    No Alert rule templates tab on the Alerting page¶

    Percona Alerting option isn’t active.

    1. Go to Configuration → Settings → Advanced Settings.
    2. Enable Alerting.

    Custom alert rule templates not migrated to Percona Alerting¶

    If you have used Integrated Alerting in previous PMM versions, and had custom templates under /srv/ia/templates, make sure to transfer them to /srv/alerting/templates. PMM is no longer sourcing templates from the ia folder, since we have deprecated Integrated Alerting with the 2.31 release.

    Unreachable external IP addresses¶

    If you get an email or page from your system that the IP is not reachable from outside my organization, do the following:

    To configure your PMM Server’s Public Address, select Configuration → Settings → Advanced Settings, and supply an address to use in your alert notifications.

    Alert Rule Templates are disabled¶

    Built-In alerts are not editable, but you can copy them and edit the copies. (In PMM 2.14.0 and above).

    If you create a custom alert rule template, you will have access to edit.

    QAN issues¶

    This section focuses on problems with QAN, such as queries not being retrieved so on.

    Missing data¶

    Why don’t I see any query-related information?

    There might be multiple places where the problem might come from:

    • Connection problem between pmm-agent and pmm-managed
    • PMM-agent cannot connect to the database.
    • Data source is not properly configured.

    Why don’t I see the whole query?

    Long query examples and fingerprints can be truncated to 1024 symbols to reduce space usage. In this case, the query explains section will not work.

    Plugins issues¶

    PMM does not allow to install, upgrade or remove plugins

    Users have encountered issues with installing, updating and removing plugins from PMM. The cause of this issue is the incorrect permissions assigned to the /srv/grafana/plugins directory. These permissions are preventing the grafana component from writing to the directory.

    Solution

    Set the ownership on the directory/srv/grafana/plugins to grafana:grafana.

    Contact us

    For free technical help, visit the Percona Community Forum.

    To report bugs or submit feature requests, open a JIRA ticket.

    For paid support and managed or consulting services , contact Percona Sales.


    Last update: 2023-09-27
    Percona LLC, © 2023
    Made with Material for MkDocs

    Cookie consent

    We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better. Read more about Percona Cookie Policy.