Skip to content
logo
Percona Server for MySQL
Jemalloc memory allocation profiling
Initializing search
    percona/psmysql-docs
    percona/psmysql-docs
    • Home
      • Release notes index
      • Percona Server for MySQL 8.0.34-26 (2023-09-26)
      • Percona Server for MySQL 8.0.33-25 Update (2023-08-02)
      • Percona Server for MySQL 8.0.33-25 (2023-06-15)
      • Percona Server for MySQL 8.0.32-24 (2023-03-20)
      • Percona Server for MySQL 8.0.31-23 (2023-02-09)
        • Percona Server for MySQL 8.0.30-22 Update (2022-11-21)
        • Percona Server for MySQL 8.0.30-22 (2022-11-21)
        • Percona Server for MySQL 8.0.29-21 (2022-08-08)
        • Percona Server for MySQL 8.0.28-20 (2022-06-20)
        • Percona Server for MySQL 8.0.28-19 (2022-05-12)
        • Percona Server for MySQL 8.0.27-18 (2022-03-02)
        • Percona Server for MySQL 8.0.26-17 (2022-01-26)
        • Percona Server for MySQL 8.0.26-16 (2021-10-20)
        • Percona Server for MySQL 8.0.25-15 (2021-07-13)
        • Percona Server for MySQL 8.0.23-14 (2021-05-12)
        • Percona Server for MySQL 8.0.22-13 (2020-12-14)
        • Percona Server for MySQL 8.0.21-12 (2020-10-13)
        • Percona Server for MySQL 8.0.20-11 (2020-07-21)
        • Percona Server for MySQL 8.0.19-10 (2020-03-23)
        • Percona Server for MySQL 8.0.18-9
        • Percona Server for MySQL 8.0.17-8
        • Percona Server for MySQL 8.0.16-7
        • Percona Server for MySQL 8.0.15-6
        • Percona Server for MySQL 8.0.15-5
        • Percona Server for MySQL 8.0.14
        • Percona Server for MySQL 8.0.13-4
        • Percona Server for MySQL 8.0.13-3
        • Percona Server for MySQL 8.0.12-2rc1
      • Adaptive network buffers
        • Audit Log Filter overview
        • Install the Audit Log Filter
          • Overview
          • XML (New style)
          • XML (Old style)
          • JSON
        • Audit Log Filter security
        • Audit Log Filter compression and encryption
        • Reading Audit Log Filter files
        • Manage the Audit Log Filter files
        • Filter the Audit Log Filter logs
        • Audit Log Filter restrictions
        • Audit Log Filter file naming conventions
        • Disable Audit Log Filter logging
        • Audit log filter functions, options and variables
        • Uninstall Audit Log Filter
      • Limiting the disk space used by binary log files
      • Extended mysqlbinlog
      • Extended SELECT INTO OUTFILE/DUMPFILE
      • Expanded fast index creation
      • Kill idle transactions
      • The ProcFS plugin
      • Support for PROXY protocol
      • SEQUENCE_TABLE(n) function
      • Slow query log rotation and expiration
      • Thread pool
      • Trigger updates
      • Percona Toolkit UDFs
      • Utility user
      • Quickstart guide for Percona Server for MySQL
      • Install Percona Server for MySQL from repositories
        • Percona Product Download Instructions
        • Use APT repositories
        • Files in DEB package
        • Build APT packages
        • Downloaded DEB packages
        • Apt pinning
        • Run Percona Server for MySQL
        • Uninstall
        • Use RPM repositories
        • Files in RPM package
        • Downloaded RPM packages
        • Run Percona Server for MySQL
        • Uninstall
        • Install with binary tarballs
        • Binary tarballs available
        • Install Percona Server for MySQL from a source tarball
        • Compile Percona Server for MySQL 8.0 from source
        • Install using Docker
        • Docker environment variables
      • Upgrade from 5.7 to 8.0 overview
      • Plan an upgrade
      • Upgrade strategies
        • General changes
        • InnoDB changes
        • Security & account management changes in MySQL 8.0
        • Deprecated in MySQL 8.0
        • Removed in MySQL 8.0
      • Percona Tools that can help with an upgrade
      • Percona Server for MySQL in-place upgrade guide: from 5.7 to 8.0
      • Upgrade by migrating from one environment to another
      • Upgrade using the Percona repositories
      • Upgrade from systems that use the MyRocks or TokuDB storage engine and partitioned tables
      • Upgrade using Standalone Packages
      • Downgrade Percona Server for MySQL
      • Working with AppArmor
      • Binary logs and replication improvements
      • Post-installation
      • Working with SELinux
      • Extended SHOW GRANTS
      • UNINSTALL COMPONENT
      • Backup and restore overview
      • Backup locks
      • Extended mysqldump
      • Start transaction with consistent snapshot
        • Using LDAP authentication plugins
        • LDAP authentication plugin system variables
        • Data masking overview
        • Compare the data masking component to the data masking plugin
          • Install the data masking component
          • Data masking component functions
          • Uninstall the data masking component
          • Install and remove the data masking plugin
          • Data masking plugin functions
      • FIDO authentication plugin
      • Encryption functions
      • PAM authentication plugin
      • SSL improvements
      • The secure_log_path variable
        • Data at Rest Encryption
        • Use the keyring component or keyring plugin
          • Using the Key Management Interoperability Protocol (KMIP)
          • Use the Amazon Key Management Service (AWS KMS)
          • Encrypt File-Per-Table Tablespace
          • Encrypt schema or general tablespace
          • Encrypt system tablespace
          • Encrypt temporary files
          • Encrypt Binary Log Files and Relay Log Files
          • Encrypting the Redo Log data
          • Encrypt the undo tablespace
          • Rotate the master key
          • Advanced encryption key rotation
          • Encrypt doublewrite buffers
          • Verify the encryption for tables, tablespaces, and schemas
      • Manage group replication flow control
      • Group replication system variables
      • Audit log plugin
      • Jemalloc memory allocation profiling
        • Requirements
        • Use Percona Server for MySQL with jemalloc with profiling enabled
        • Dumping the profile
        • PERFORMANCE_SCHEMA tables
        • malloc_stats_totals
        • malloc_stats
        • System variables
          • jemalloc_detected
          • jemalloc_profiling
        • Disable profiling
      • Misc. INFORMATION_SCHEMA tables
      • Process list
      • Slow query log
      • User statistics
      • Use Percona Monitoring and Management (PMM) Advisors
      • Handle corrupted tables
      • Libcoredumper
      • Too many connections warning
      • Stacktrace
      • Thread based profiling
        • Multiple page asynchronous I/O requests
        • XtraDB changed page tracking
        • Compressed columns with dictionaries
        • Enforcing storage engine
        • Improved MEMORY storage engine
        • InnoDB page fragmentation counters
        • InnoDB full-text search improvements
        • Improved InnoDB I/O scalability
        • Extended show engine InnoDB status
        • The Percona XtraDB storage engine
        • Prefix index queries optimization
        • Limit the estimation of records in a Query
        • Show storage engines
        • XtraDB performance improvements for I/O-bound highly-concurrent workloads
        • Percona MyRocks introduction
        • Percona MyRocks installation guide
        • Updated supported features
        • MyRocks limitations
        • Differences between Percona MyRocks and Facebook MyRocks
        • MyRocks column families
        • Performance Schema MyRocks changes
        • MyRocks Information Schema tables
        • MyRocks server variables
        • MyRocks status variables
        • Gap locks detection
        • MyRocks data loading
        • Installing and configuring Percona Server for MySQL with ZenFS support
        • TokuDB introduction
        • Get started with TokuDB
        • TokuDB installation
        • Use TokuDB
        • Fast updates with TokuDB
        • TokuDB files and file types
        • TokuDB file management
        • TokuDB background ANALYZE TABLE
        • TokuDB variables
        • TokuDB status variables
        • TokuDB fractal tree indexing
        • TokuDB troubleshooting
        • TokuDB Performance Schema integration
        • TokuDB frequently asked questions
        • Migrate and remove the TokuDB storage engine
        • Percona TokuBackup
      • Topic index
      • Reserved keywords
      • List of variables introduced in Percona Server for MySQL 8.0
      • List of features available in Percona Server for MySQL releases
      • Percona Server for MySQL feature comparison
      • Understand version numbers
      • Development of Percona Server for MySQL
      • Trademark policy
      • Index of INFORMATION_SCHEMA tables
      • Frequently asked questions
      • Copyright and licensing information
      • Glossary

    • Requirements
    • Use Percona Server for MySQL with jemalloc with profiling enabled
    • Dumping the profile
    • PERFORMANCE_SCHEMA tables
    • malloc_stats_totals
    • malloc_stats
    • System variables
      • jemalloc_detected
      • jemalloc_profiling
    • Disable profiling

    Jemalloc memory allocation profiling¶

    Implemented in Percona Server for MySQL 8.0.25-15, Percona Server for MySQL can take advantage of the memory-profiling ability of the jemalloc allocator. This ability provides a method to investigate memory-related issues.

    Requirements¶

    This memory-profiling requires jemalloc_detected. This read-only variable returns true if jemalloc with the profiling-enabled option is being used by Percona Server for MySQL.

    As root, customize jemalloc with the following flags:

    Option Description
    –enable-stats Enables statistics-gathering ability
    –enable-prof Enables heap profiling and the ability to detect leaks.

    Using LD_PRELOAD. Build the library, configure the malloc configuration with the prof:true string, and then use LD_PRELOAD to preload the libjemalloc.so library. The libprocess MemoryProfiler class detects the library automatically and enables the profiling support.

    The following is an example of the required commands:

    ./configure --enable-stats --enable-prof && make && make install
    MALLOC_CONF=prof:true
    LD_PRELOAD=/usr/lib/libjemalloc.so
    

    Use Percona Server for MySQL with jemalloc with profiling enabled¶

    To detect if jemalloc is set, run the following command:

    SELECT @@jemalloc_detected;
    

    To enable jemalloc profiling in a MySQL client, run the following command:

    set global jemalloc_profiling=on;
    

    The malloc_stats_totals table returns the statistics, in bytes, of the memory usage. The command takes no parameters and returns the results as a table.

    The following example commands display this result:

    use performance_schema;
    
    SELECT * FROM malloc_stats_totals;
    
    Expected output
    +----+------------+------------+------------+-------------+------------+
    | id | ALLOCATION | MAPPED     | RESIDENT   | RETAINED    | METADATA   |
    +----+------------+------------+------------+-------------+------------+
    |  1 | 390977528  | 405291008  | 520167424  | 436813824   | 9933744    |
    +----+------------+------------+------------+-------------+------------+
    1 row in set (0.00 sec)
    

    The malloc_stats table returns the cumulative totals, in bytes, of several statistics per type of arena. The command takes no parameters and returns the results as a table.

    The following example commands display this result:

    use performance_schema;
    
    mysql> SELECT * FROM malloc_stats ORDER BY TYPE DESC LIMIT 3;
    
    Expected output
    +--------+-------------+-------------+-------------+-------------+
    | TYPE   | ALLOCATED   | NMALLOC     | NDALLOC     | NRESQUESTS  |
    +--------+-------------+-------------+-------------+-------------+
    | small  | 23578872    | 586156      | 0           | 2649417     |
    | large  | 367382528   | 2218        | 0           | 6355        |
    | huge   | 0           | 0           | 0           | |
    +--------+-------------+-------------+-------------+-------------+
    3 rows in set (0.00 sec)
    

    Dumping the profile¶

    The profiling samples the malloc() calls and stores the sampled stack traces in a separate location in memory. These samples can be dumped into the filesystem. A dump returns a detailed view of the state of the memory.

    The process is global; therefore, only a single concurrent run is available and only the most recent runs are stored on disk.

    Use the following command to create a profile dump file:

    flush memory profile;
    

    The generated memory profile dumps are written to the /tmp directory.

    You can analyze the dump files with jeprof program, which must be installed on the host system in the appropriate path. This program is a perl script that post-processes the dump files in their raw format. The program has no connection to the jemalloc library and the version numbers are not required to match.

    To verify the dump, run the following command:

    ls /tmp/jeprof_mysqld*
    /tmp/jeprof_mysqld.1.0.170013202213
    jeprof --show_bytes /tmp/jeprof_mysqld.1.0.170013202213 jeprof.*.heap
    

    You can also access the memory profile to plot a graph of the memory use. This ability requires that jeprof and dot are in the /tmp path. For the graph to display useful information, the binary file must contain symbol information.

    Run the following command:

    jeprof --dot /usr/sbin/mysqld /tmp/jeprof_mysqld.1.0.170013202213 > /tmp/jeprof1.dot
    dot --Tpng /tmp/jeprof1.dot > /tmp/jeprof1.png
    

    Note

    An example of allocation graph.

    PERFORMANCE_SCHEMA tables¶

    In 8.0.25.14, the following tables are implemented to retrieve memory allocation statistics for a running instance or return the cumulative number of allocations requested or allocations returned for a running instance.

    More information about the stats that are returned can be found in jemalloc.

    malloc_stats_totals¶

    The current stats for allocations. All measurements are in bytes.

    Column Name Description
    ALLOCATED The total amount the application allocated
    ACTIVE The total amount allocated by the application of active pages. A multiple of the page size and this value is greater than or equal to the stats.allocated value. The sum does not include allocator metadata pages and stats.arenas.<i>.pdirty or stats.arenas.<i>.pmuzzy.
    MAPPED The total amount in chunks that are mapped by the allocator in active extents. This value does not include inactive chunks. The value is at least as large as the stats.active and is a multiple of the chunk size.
    RESIDENT A maximum number the allocator has mapped in physically resident data pages. All allocator metadata pages and unused dirty pages are included in this value. Pages may not be physically resident if they correspond to demand-zeroed virtual memory that has not yet been touched. This value is a maximum rather than a precise value and is a multiple of the page size. The value is greater than the stats.active.
    RETAINED The amount retained by the virtual memory mappings of the operating system. This value does not include any returned mappings. This type of memory, usually de-committed, untouched, or purged. The value is associated with physical memory and is excluded from mapped memory statistics.
    METADATA The total amount dedicated to metadata. This value contains the base allocations which are used for bootstrap-sensitive allocator metadata structures. Transparent huge pages usage is not included.

    malloc_stats¶

    The cumulative number of allocations requested or allocations returned for a running instance.

    Column Name Description
    Type The type of object: small, large, and huge
    ALLOCATED The number of bytes that are currently allocated to the application.
    NMALLOC A cumulative number of times an allocation was requested from the arena’s bins. The number includes times when the allocation satisfied an allocation request or filled a relevant tcache if opt.tcache is enabled.
    NDALLOC A cumulative number of times an allocation was returned to the arena’s bins. The number includes times when the allocation was deallocated or flushed the relevant tcache if opt.tcache is enabled.
    NREQUESTS The cumulative number of allocation requests satisfied.

    System variables¶

    The following variables have been added:

    jemalloc_detected¶

    Description: This read-only variable returns true if jemalloc with profiling enabled is detected. The following options are required:

    • Jemalloc is installed and compiled with profiling enabled

    • Percona Server for MySQL is configured to use jemalloc by using the environment variable LD_PRELOAD.

    • The environment variable MALLOC_CONF is set to prof:true.

    The following options are:

    • Scope: Global

    • Variable Type: Boolean

    • Default Value: false

    jemalloc_profiling¶

    Description: Enables jemalloc profiling. The variable requires jemalloc_detected.

    • Command Line: –jemalloc_profiling[=(OFF|ON)]

    • Config File: Yes

    • Scope: Global

    • Dynamic: Yes

    • Variable Type: Boolean

    • Default Value: OFF

    Disable profiling¶

    To disable jemalloc profiling, in a MySQL client, run the following command:

    set global jemalloc_profiling=off;
    

    Contact us

    For free technical help, visit the Percona Community Forum.

    To report bugs or submit feature requests, open a JIRA ticket.

    For paid support and managed or consulting services , contact Percona Sales.


    Last update: 2023-09-27
    Percona LLC and/or its affiliates, © 2023
    Made with Material for MkDocs

    Cookie consent

    We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better. Read more about Percona Cookie Policy.