Skip to content
logo
Percona XtraBackup
Smart memory estimation
Initializing search
    percona/pxb-docs
    percona/pxb-docs
    • Home
      • Release notes index
      • Percona XtraBackup 8.0.35-30 (2023-12-04)
      • Percona XtraBackup 8.0.34-29 (2023-08-21)
      • Percona XtraBackup 8.0.33-28 (2023-07-19)
      • Percona XtraBackup 8.0.33-27 (2023-05-25)
      • Percona XtraBackup 8.0.32-26 (2023-04-04)
      • Percona XtraBackup 8.0.32-25 (2023-02-27)
      • Percona XtraBackup 8.0.31-24 (2023-02-07)
        • Percona XtraBackup 8.0.30-23 (2022-11-14)
        • Percona XtraBackup 8.0.29-22 (2022-07-19)
        • Percona XtraBackup 8.0.28-21 (2022-05-25)
        • Percona XtraBackup 8.0.28-20
        • Percona XtraBackup 8.0.27-19
        • Percona XtraBackup 8.0.26-18.0
        • Percona XtraBackup 8.0.25-17.0
        • Percona XtraBackup 8.0.23-16.0
        • Percona XtraBackup 8.0.22-15.0
        • Percona XtraBackup 8.0.14
        • Percona XtraBackup 8.0.13
        • Percona XtraBackup 8.0.12
        • Percona XtraBackup 8.0.11
        • Percona XtraBackup 8.0.10
        • Percona XtraBackup 8.0.9
        • Percona XtraBackup 8.0.8
        • Percona XtraBackup 8.0.7
        • Percona XtraBackup 8.0.6
        • Percona XtraBackup 8.0.5
        • Percona XtraBackup 8.0.4
        • Percona XtraBackup 8.0-3-rc1
      • About Percona XtraBackup
      • How Percona XtraBackup works
      • Understand version numbers
        • LRU dump backup
        • Throttling backups
        • Store backup history on the server
        • Dictionary cache
        • Point-in-time recovery
        • Restore individual tables
      • Encrypted InnoDB tablespace backups
      • FLUSH TABLES WITH READ LOCK option
      • Improved log statements
      • lock-ddl-per-table option improvements
      • Smart memory estimation
        • How does Smart memory estimation work
        • Example of Smart memory estimation usage
      • Work with binary logs
      • Overview
        • The xtrabackup binary overview
        • xtrabackup implementation details
        • Configure xtrabackup
        • Analyze table statistics
      • The xbcrypt binary overview
        • The xbstream binary overview
        • Take a streaming backup
        • Accelerate the backup process
        • Encrypt backups
        • The xbcloud binary overview
          • Use the xbcloud binary with Amazon S3
          • Use the xbcloud binary with an IAM instance profile
          • Use the xbcloud binary with Swift
          • Use the xbcloud binary with Google Cloud Storage
          • Use the xbcloud binary with Microsoft Azure Cloud Storage
          • Use the xbcloud binary with MinIO
        • Update curl utility
        • FIFO data sink
        • Exponential backoff
      • Quickstart Guide for Percona XtraBackup
      • Install Percona XtraBackup 8.0 overview
        • Server version and backup version comparison
        • Connection and privileges needed
        • Permissions needed
        • Use APT repositories
        • Files in DEB package
        • Downloaded DEB packages
        • Apt pinning
        • Work with AppArmor
        • Uninstall
        • Use RPM repositories
        • Files in RPM package
        • Downloaded RPM packages
        • Work with SELinux
        • Uninstall
        • Install with binary tarballs
        • Binary tarballs available
        • Compile and install Percona XtraBackup 8.0 from source
        • Run Percona XtraBackup 8.0 in a Docker container
        • Create a full backup
        • Prepare a full backup
        • Create an incremental backup
        • Prepare an incremental backup
        • Take an incremental backup using page tracking
        • Create a compressed backup
        • Decompress and prepare a compressed backup
        • Create a partial backup
        • Prepare a partial backup
        • Create an individual partitions backup
        • Prepare an individual partitions backup
        • Restore full, incremental, compressed backups
        • Restore a partial backup
        • Restore an individual partitions backup
        • How to set up a replica for replication in 6 simple steps with Percona XtraBackup
        • How to create a new (or repair a broken) GTID-based replica
        • Make backups in replication environments
        • Verify backups with replication and pt-checksum
      • Error Message: Found tables with row versions due to INSTANT ADD/DROP columns
      • xtrabackup exit codes
      • Topic index
      • The xtrabackup option reference
      • Index of files created by Percona XtraBackup
      • Frequently asked questions
      • Glossary
      • Percona Toolkit version checking
      • Trademark policy
      • Copyright and licensing information

    • How does Smart memory estimation work
    • Example of Smart memory estimation usage

    Smart memory estimation¶

    The Smart memory estimation is tech preview feature. Before using Smart memory estimation in production, we recommend that you test restoring production from physical backups in your environment and also use the alternative backup method for redundancy.

    Percona XtraBackup 8.0.30-23 adds support for the Smart memory estimation feature. With this feature, Percona XtraBackup computes the memory required for prepare phase, while copying redo log entries during the backup phase. Percona XtraBackup also considers the number of InnoDB pages to be fetched from the disk.

    Percona XtraBackup performs the backup procedure in two steps:

    • Creates a backup

      To create a backup, Percona XtraBackup copies your InnoDB data files. While copying the files, Percona XtraBackup runs a background process that watches the InnoDB redo log, also called the transaction log, and copies changes from it.

    • Prepares a backup

      During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files using the copied transaction log file. Percona XtraBackup reads all the redo log entries into memory, categorizes them by space id and page id, reads the relevant pages into memory, and checks the log sequence number (LSN) on the page and on the redo log record. If the redo log LSN is more recent than the page LSN, Percona XtraBackup applies the redo log changes to the page.

      To prepare a backup, Percona Xtrabackup uses InnoDB Buffer Pool memory. Percona Xtrabackup reserves memory to load 256 pages into the buffer pool. The remaining memory is used for hashing/categorizing the redo log entries.

      The available memory is controlled by the --use-memory option. If the available memory on the buffer pool is insufficient, the work is performed in multiple batches. After the batch is processed, the memory is freed to release space for the next batch. This process greatly impacts performance as an InnoDB page holds data from multiple rows. If a change on a page happens in different batches, that page is fetched and evicted numerous times.

    How does Smart memory estimation work¶

    In the prepare phase, Percona XtraBackup checks the server’s available free memory and uses that memory up to the limit specified in the --use-free-memory-pct option to run --prepare. Due to backward compatibility, the default value for the --use-free-memory-pct option is 0 (zero), which defines the option as disabled. For example, if you set --use-free-memory-pct=50, then 50% of the free memory is used to prepare a backup.

    Starting with Percona XtraBackup 8.0.32-26, you can enable or disable the memory estimation during the backup phase with the --estimate-memory option. The default value is OFF. Enable the memory estimation with --estimate-memory=ON:

    $ xtrabackup --backup --estimate-memory=ON --target-dir=/data/backups/
    

    In the prepare phase, enable the --use-free-memory-pct option by specifying the percentage of free memory to be used to prepare a backup. The --use-free-memory-pct value must be larger than 0.

    For example:

    $ xtrabackup --prepare --use-free-memory-pct=50 --target-dir=/data/backups/
    

    Example of Smart memory estimation usage¶

    The examples of how Smart memory estimation can improve the time spent on prepare in different versions of Percona XtraBackup:

    We back up 16, 32, and 64 tables using sysbench. Each set contains 1M rows. In the backup phase, we enable Smart memory estimation with --estimate-memory=ON. In the prepare phase, we set --use-free-memory-pct=50, and Percona XtraBackup uses 50% of the free memory to prepare a backup. The backup is run on an ec2 c4.8xlarge instance (36 vCPUs / 60GB memory / General Purpose SSD (gp2)).

    We back up 16, 32, and 64 tables using sysbench. Each set contains 1M rows. In the prepare phase, we set --use-free-memory-pct=50, and Percona XtraBackup uses 50% of the free memory to prepare a backup. The backup is run on an ec2 c4.8xlarge instance (36 vCPUs / 60GB memory / General Purpose SSD (gp2)).

    During each --backup, the following sysbench is run:

    sysbench --db-driver=mysql --db-ps-mode=disable --mysql-user=sysbench --mysql-password=sysbench --table_size=1000000 --tables=${NUM_OF_TABLES} --threads=24 --time=0 --report-interval=1 /usr/share/sysbench/oltp_write_only.lua run
    

    The following table shows the backup details (all measurements are in Gigabytes):

    Used memory Size of XtraBackup log Size of backup
    16 tables 3.375 0.7 4.7
    32 tables 8.625 2.6 11
    64 tables 18.5 5.6 22
    • Used memory - the amount of memory required by Percona XtraBackup with --use-free-memory-pct=50

    • Size of XtraBackup log - the size of Percona XtraBackup log file (redo log entries copied during the backup)

    • Size of backup - the size of the resulting backup folder

    Prepare executed without Smart memory estimation uses the default of 128MB for the buffer pool.

    The results are the following:

    Note

    The following results are based on tests in a specific environment. Your results may vary.

    Time to run --prepare

    • 16 tables result - prepare time dropped to ~5.7% of the original time. An improvement in recovery time of about 17x.

    • 32 tables result - prepare time dropped to ~8,2% of the original time. An improvement in recovery time of about 12x.

    • 64 tables result - prepare time dropped to ~9.9% of the original time. An improvement in recovery time of about 10x.

    Contact us

    For free technical help, visit the Percona Community Forum.

    To report bugs or submit feature requests, open a JIRA ticket.

    For paid support and managed or consulting services , contact Percona Sales.


    Last update: 2023-10-26
    Percona LLC and/or its affiliates, © 2023
    Made with Material for MkDocs

    Cookie consent

    We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better. Read more about Percona Cookie Policy.