Skip to content
logo
Percona Server for MySQL
Jemalloc memory allocation profiling
Initializing search
    percona/psmysql-docs
    percona/psmysql-docs
    • Home
      • The Percona XtraDB storage engine
      • List of features available in Percona Server for MySQL releases
      • Percona Server for MySQL feature comparison
      • Changed in Percona Server 8.0
      • Understand version numbers
      • Install Percona Server for MySQL
        • Install using APT repository
        • Files in DEB package
        • Build APT packages
        • Install from downloaded DEB packages
        • Apt pinning
        • Run Percona Server for MySQL
        • Uninstall
        • Install using a RPM repository
        • Files in RPM package
        • Install from downloaded RPM packages
        • Run Percona Server for MySQL
        • Uninstall
        • Install with binary tarballs
        • Binary tarballs available
        • Install Percona Server for MySQL from a source tarball
        • Compile Percona Server for MySQL 8.0 from source
      • Post-installation
      • Percona Server for MySQL in-place upgrading guide: from 5.7 to 8.0
      • Upgrade using the Percona repositories
      • Upgrade from systems that use the MyRocks or TokuDB storage engine and partitioned tables
      • Upgrade using Standalone Packages
      • Downgrade Percona Server for MySQL
      • Running Percona Server for MySQL in a Docker Container
      • Docker environment variables
      • Improved InnoDB I/O scalability
      • Adaptive network buffers
      • Multiple page asynchronous I/O requests
      • Thread pool
      • XtraDB performance improvements for I/O-bound highly-concurrent workloads
      • Prefix index queries optimization
      • Limit the estimation of records in a Query
      • Jemalloc memory allocation profiling
        • Requirements
        • Use Percona Server for MySQL with jemalloc with profiling enabled
        • Dumping the profile
        • PERFORMANCE_SCHEMA tables
        • malloc_stats_totals
        • malloc_stats
        • System variables
          • jemalloc_detected
          • jemalloc_profiling
        • Disable profiling
      • The ProcFS plugin
      • Binary logs and replication improvements
      • Compressed columns with dictionaries
      • Extended SELECT INTO OUTFILE/DUMPFILE
      • Extended SET VAR optimizer hint
      • Improved MEMORY storage engine
      • Suppress warning messages
      • Limiting the disk space used by binary log files
      • Support for PROXY protocol
      • SEQUENCE_TABLE(n) function
      • Slow query log rotation and expiration
      • Trigger updates
      • Extended mysqlbinlog
      • Extended mysqldump
      • InnoDB full-text search improvements
      • Too many connections warning
      • Handle corrupted tables
      • Percona Toolkit UDFs
      • Kill idle transactions
      • XtraDB changed page tracking
      • Enforcing storage engine
      • Expanded fast index creation
      • Backup locks
      • Audit log plugin
      • Start transaction with consistent snapshot
      • Extended SHOW GRANTS
      • Utility user
      • Working with SELinux
      • Working with AppArmor
      • PAM authentication plugin
      • Server variables
      • SSL improvements
      • Data masking
        • Using LDAP authentication plugins
        • LDAP authentication plugin system variables
        • Data at Rest Encryption
        • Use the keyring component or keyring plugin
          • Using the Key Management Interoperability Protocol (KMIP)
          • Use the Amazon Key Management Service (AWS KMS)
          • FIDO authentication plugin
        • Encryption functions
        • Encrypt File-Per-Table Tablespace
        • Encrypt schema or general tablespace
        • Encrypt system tablespace
        • Encrypt temporary files
        • Encrypt Binary Log Files and Relay Log Files
        • Encrypting the Redo Log data
        • Encrypt the undo tablespace
        • Rotate the master key
        • Advanced encryption key rotation
        • Encrypt doublewrite buffers
        • Verify the encryption for tables, tablespaces, and schemas
      • User statistics
      • Slow query log
      • Extended show engine InnoDB status
      • Show storage engines
      • Process list
      • Misc. INFORMATION_SCHEMA tables
      • Thread based profiling
      • InnoDB page fragmentation counters
      • Stacktrace
      • Libcoredumper
      • Manage group replication flow control
      • Group replication system variables
      • Percona MyRocks introduction
      • Percona MyRocks installation guide
      • Updated supported features
      • MyRocks limitations
      • Differences between Percona MyRocks and Facebook MyRocks
      • MyRocks Information Schema tables
      • MyRocks server variables
      • MyRocks status variables
      • Gap locks detection
      • Data loading
      • Installing and configuring Percona Server for MySQL with ZenFS support
      • TokuDB introduction
      • TokuDB installation
      • Use TokuDB
      • Fast updates with TokuDB
      • TokuDB files and file types
      • TokuDB file management
      • TokuDB background ANALYZE TABLE
      • TokuDB variables
      • TokuDB status variables
      • TokuDB fractal tree indexing
      • TokuDB troubleshooting
      • TokuDB Performance Schema integration
      • Frequently asked questions
      • Migrate and removing the TokuDB storage engine
      • Percona TokuBackup
      • Release notes index
      • Percona Server for MySQL 8.0.32-24 (2023-03-20)
      • Percona Server for MySQL 8.0.31-23 (2023-02-09)
      • Percona Server for MySQL 8.0.30-22 Update (2022-11-21)
      • Percona Server for MySQL 8.0.30-22 (2022-11-21)
      • Percona Server for MySQL 8.0.29-21 (2022-08-08)
      • Percona Server for MySQL 8.0.28-20 (2022-06-20)
      • Percona Server for MySQL 8.0.28-19 (2022-05-12)
      • Percona Server for MySQL 8.0.27-18 (2022-03-02)
      • Percona Server for MySQL 8.0.26-17 (2022-01-26)
      • Percona Server for MySQL 8.0.26-16 (2021-10-20)
      • Percona Server for MySQL 8.0.25-15 (2021-07-13)
      • Percona Server for MySQL 8.0.23-14 (2021-05-12)
      • Percona Server for MySQL 8.0.22-13 (2020-12-14)
      • Percona Server for MySQL 8.0.21-12 (2020-10-13)
      • Percona Server for MySQL 8.0.20-11 (2020-07-21)
      • Percona Server for MySQL 8.0.19-10 (2020-03-23)
      • Percona Server for MySQL 8.0.18-9
      • Percona Server for MySQL 8.0.17-8
      • Percona Server for MySQL 8.0.16-7
      • Percona Server for MySQL 8.0.15-6
      • Percona Server for MySQL 8.0.15-5
      • Percona Server for MySQL 8.0.14
      • Percona Server for MySQL 8.0.13-4
      • Percona Server for MySQL 8.0.13-3
      • Percona Server for MySQL 8.0.12-2rc1
      • List of variables introduced in Percona Server for MySQL 8.0
      • Development of Percona Server for MySQL
      • Trademark policy
      • Index of INFORMATION_SCHEMA tables
      • Frequently asked questions
      • Copyright and licensing information
      • Glossary

    • Requirements
    • Use Percona Server for MySQL with jemalloc with profiling enabled
    • Dumping the profile
    • PERFORMANCE_SCHEMA tables
    • malloc_stats_totals
    • malloc_stats
    • System variables
      • jemalloc_detected
      • jemalloc_profiling
    • Disable profiling

    Jemalloc memory allocation profiling¶

    Implemented in Percona Server for MySQL 8.0.25-15, Percona Server for MySQL can take advantage of the memory-profiling ability of the jemalloc allocator. This ability provides a method to investigate memory-related issues.

    Requirements¶

    This memory-profiling requires jemalloc_detected. This read-only variable returns true if jemalloc with the profiling-enabled option is being used by Percona Server for MySQL.

    As root, customize jemalloc with the following flags:

    Option Description
    –enable-stats Enables statistics-gathering ability
    –enable-prof Enables heap profiling and the ability to detect leaks.

    Using LD_PRELOAD. Build the library, configure the malloc configuration with the prof:true string, and then use LD_PRELOAD to preload the libjemalloc.so library. The libprocess MemoryProfiler class detects the library automatically and enables the profiling support.

    The following is an example of the required commands:

    ./configure --enable-stats --enable-prof && make && make install
    MALLOC_CONF=prof:true
    LD_PRELOAD=/usr/lib/libjemalloc.so
    

    Use Percona Server for MySQL with jemalloc with profiling enabled¶

    To detect if jemalloc is set, run the following command:

    SELECT @@jemalloc_detected;
    

    To enable jemalloc profiling in a MySQL client, run the following command:

    set global jemalloc_profiling=on;
    

    The malloc_stats_totals table returns the statistics, in bytes, of the memory usage. The command takes no parameters and returns the results as a table.

    The following example commands display this result:

    use performance_schema;
    
    SELECT * FROM malloc_stats_totals;
    
    Expected output
    +----+------------+------------+------------+-------------+------------+
    | id | ALLOCATION | MAPPED     | RESIDENT   | RETAINED    | METADATA   |
    +----+------------+------------+------------+-------------+------------+
    |  1 | 390977528  | 405291008  | 520167424  | 436813824   | 9933744    |
    +----+------------+------------+------------+-------------+------------+
    1 row in set (0.00 sec)
    

    The malloc_stats table returns the cumulative totals, in bytes, of several statistics per type of arena. The command takes no parameters and returns the results as a table.

    The following example commands display this result:

    use performance_schema;
    
    mysql> SELECT * FROM malloc_stats ORDER BY TYPE DESC LIMIT 3;
    
    Expected output
    +--------+-------------+-------------+-------------+-------------+
    | TYPE   | ALLOCATED   | NMALLOC     | NDALLOC     | NRESQUESTS  |
    +--------+-------------+-------------+-------------+-------------+
    | small  | 23578872    | 586156      | 0           | 2649417     |
    | large  | 367382528   | 2218        | 0           | 6355        |
    | huge   | 0           | 0           | 0           | |
    +--------+-------------+-------------+-------------+-------------+
    3 rows in set (0.00 sec)
    

    Dumping the profile¶

    The profiling samples the malloc() calls and stores the sampled stack traces in a separate location in memory. These samples can be dumped into the filesystem. A dump returns a detailed view of the state of the memory.

    The process is global; therefore, only a single concurrent run is available and only the most recent runs are stored on disk.

    Use the following command to create a profile dump file:

    flush memory profile;
    

    The generated memory profile dumps are written to the /tmp directory.

    You can analyze the dump files with jeprof program, which must be installed on the host system in the appropriate path. This program is a perl script that post-processes the dump files in their raw format. The program has no connection to the jemalloc library and the version numbers are not required to match.

    To verify the dump, run the following command:

    ls /tmp/jeprof_mysqld*
    /tmp/jeprof_mysqld.1.0.170013202213
    jeprof --show_bytes /tmp/jeprof_mysqld.1.0.170013202213 jeprof.*.heap
    

    You can also access the memory profile to plot a graph of the memory use. This ability requires that jeprof and dot are in the /tmp path. For the graph to display useful information, the binary file must contain symbol information.

    Run the following command:

    jeprof --dot /usr/sbin/mysqld /tmp/jeprof_mysqld.1.0.170013202213 > /tmp/jeprof1.dot
    dot --Tpng /tmp/jeprof1.dot > /tmp/jeprof1.png
    

    Note

    An example of allocation graph.

    PERFORMANCE_SCHEMA tables¶

    In 8.0.25.14, the following tables are implemented to retrieve memory allocation statistics for a running instance or return the cumulative number of allocations requested or allocations returned for a running instance.

    More information about the stats that are returned can be found in jemalloc.

    malloc_stats_totals¶

    The current stats for allocations. All measurements are in bytes.

    Column Name Description
    ALLOCATED The total amount the application allocated
    ACTIVE The total amount allocated by the application of active pages. A multiple of the page size and this value is greater than or equal to the stats.allocated value. The sum does not include allocator metadata pages and stats.arenas.<i>.pdirty or stats.arenas.<i>.pmuzzy.
    MAPPED The total amount in chunks that are mapped by the allocator in active extents. This value does not include inactive chunks. The value is at least as large as the stats.active and is a multiple of the chunk size.
    RESIDENT A maximum number the allocator has mapped in physically resident data pages. All allocator metadata pages and unused dirty pages are included in this value. Pages may not be physically resident if they correspond to demand-zeroed virtual memory that has not yet been touched. This value is a maximum rather than a precise value and is a multiple of the page size. The value is greater than the stats.active.
    RETAINED The amount retained by the virtual memory mappings of the operating system. This value does not include any returned mappings. This type of memory, usually de-committed, untouched, or purged. The value is associated with physical memory and is excluded from mapped memory statistics.
    METADATA The total amount dedicated to metadata. This value contains the base allocations which are used for bootstrap-sensitive allocator metadata structures. Transparent huge pages usage is not included.

    malloc_stats¶

    The cumulative number of allocations requested or allocations returned for a running instance.

    Column Name Description
    Type The type of object: small, large, and huge
    ALLOCATED The number of bytes that are currently allocated to the application.
    NMALLOC A cumulative number of times an allocation was requested from the arena’s bins. The number includes times when the allocation satisfied an allocation request or filled a relevant tcache if opt.tcache is enabled.
    NDALLOC A cumulative number of times an allocation was returned to the arena’s bins. The number includes times when the allocation was deallocated or flushed the relevant tcache if opt.tcache is enabled.
    NREQUESTS The cumulative number of allocation requests satisfied.

    System variables¶

    The following variables have been added:

    jemalloc_detected¶

    Description: This read-only variable returns true if jemalloc with profiling enabled is detected. The following options are required:

    • Jemalloc is installed and compiled with profiling enabled

    • Percona Server for MySQL is configured to use jemalloc by using the environment variable LD_PRELOAD.

    • The environment variable MALLOC_CONF is set to prof:true.

    The following options are:

    • Scope: Global

    • Variable Type: Boolean

    • Default Value: false

    jemalloc_profiling¶

    Description: Enables jemalloc profiling. The variable requires jemalloc_detected.

    • Command Line: –jemalloc_profiling[=(OFF|ON)]

    • Config File: Yes

    • Scope: Global

    • Dynamic: Yes

    • Variable Type: Boolean

    • Default Value: OFF

    Disable profiling¶

    To disable jemalloc profiling, in a MySQL client, run the following command:

    set global jemalloc_profiling=off;
    

    Contact us

    For free technical help, visit the Percona Community Forum.

    To report bugs or submit feature requests, open a JIRA ticket.

    For paid support and managed or consulting services , contact Percona Sales.


    Last update: 2023-01-12
    Percona LLC and/or its affiliates, © 2023
    Made with Material for MkDocs

    Cookie consent

    We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better.