Skip to content
Starting November 2023 Percona XtraBackup 2.4 has reached EOL status. If you have 5.7 databases, we encourage you to upgrade to 8.0 and then install Percona XtraBackup 8.0. Learn more

logo
Percona XtraBackup
The xbcloud Binary
Initializing search
    percona/pxb-docs
    percona/pxb-docs
    • Home
      • About Percona XtraBackup
      • How Percona XtraBackup Works
      • Understand version numbers
      • Installing Percona XtraBackup 2.4
      • Installing Percona XtraBackup on Debian and Ubuntu
      • Installing Percona XtraBackup on Red Hat Enterprise Linux and CentOS
      • Installing Percona XtraBackup from a Binary Tarball
      • Compiling and Installing from Source Code
      • Running Percona XtraBackup in a Docker container
      • Connection and Privileges Needed
      • Configuring xtrabackup
      • The Backup Cycle - Full Backups
      • Incremental Backup
      • Compressed Backup
      • Encrypted Backup
      • Percona XtraBackup User Manual
      • Throttling Backups
      • Lockless binary log information
      • Encrypted InnoDB Tablespace Backups
      • lock-ddl-per-table Option Improvements
      • How-tos and Recipes
        • Release notes index
        • Percona XtraBackup 2.4.29 (2023-12-18)
        • Percona XtraBackup 2.4.28 (2023-04-04)
        • Percona XtraBackup 2.4.27 (2022-12-06)
        • Percona XtraBackup 2.4.26
        • Percona XtraBackup 2.4.25
        • Percona XtraBackup 2.4.24
        • Percona XtraBackup 2.4.23
        • Percona XtraBackup 2.4.22
        • Percona XtraBackup 2.4.21
        • Percona XtraBackup 2.4.20
        • Percona XtraBackup 2.4.19
        • Percona XtraBackup 2.4.18
        • Percona XtraBackup 2.4.17
        • Percona XtraBackup 2.4.16
        • Percona XtraBackup 2.4.15
        • Percona XtraBackup 2.4.14
        • Percona XtraBackup 2.4.13
        • Percona XtraBackup 2.4.12
        • Percona XtraBackup 2.4.11
        • Percona XtraBackup 2.4.10
        • Percona XtraBackup 2.4.9
        • Percona XtraBackup 2.4.8
        • Percona XtraBackup 2.4.7-2
        • Percona XtraBackup 2.4.7
        • Percona XtraBackup 2.4.6
        • Percona XtraBackup 2.4.5
        • Percona XtraBackup 2.4.4
        • Percona XtraBackup 2.4.3
        • Percona XtraBackup 2.4.2
        • Percona XtraBackup 2.4.1
      • The xtrabackup Option Reference
      • The innobackupex Option Reference
      • The xbcloud Binary
        • Version specific information
        • Supported Cloud Storage Types
        • Usage
        • Creating a full backup with Swift
        • Creating a full backup with Amazon S3
        • Creating a full backup with MinIO
        • Creating a full backup with Google Cloud Storage
        • Supplying parameters
          • Configuration files
          • Environment variables
          • Shortcuts
          • Additional parameters
        • Restoring with Swift
        • Restoring with Amazon S3
        • Incremental backups
          • Preparing an incremental backup
          • Partial download of the cloud backup
        • Command-line options
          • –storage=[swiftAmazon S3google]
          • –swift-auth-url
          • –swift-url
          • –swift-storage-url
          • –swift-user
          • –swift-key
          • –swift-container
          • –parallel=N
          • –cacert
          • –insecure
          • Swift authentication options
          • –swift-auth-version
          • –swift-tenant
          • –swift-tenant-id
          • –swift-region
          • –swift-password
          • –swift-user-id
          • –swift-project
          • –swift-project-id
          • –swift-domain
          • –swift-domain-id
      • Exponential Backoff
      • Using the xbcloud binary with Microsoft Azure Cloud Storage
      • The xbcrypt binary
      • The xbstream binary
      • Known issues and limitations
      • Frequently Asked Questions
      • Glossary
      • Index of files created by Percona XtraBackup
      • Trademark policy
      • Copyright and licensing information
      • Version Checking

    • Version specific information
    • Supported Cloud Storage Types
    • Usage
    • Creating a full backup with Swift
    • Creating a full backup with Amazon S3
    • Creating a full backup with MinIO
    • Creating a full backup with Google Cloud Storage
    • Supplying parameters
      • Configuration files
      • Environment variables
      • Shortcuts
      • Additional parameters
    • Restoring with Swift
    • Restoring with Amazon S3
    • Incremental backups
      • Preparing an incremental backup
      • Partial download of the cloud backup
    • Command-line options
      • –storage=[swiftAmazon S3google]
      • –swift-auth-url
      • –swift-url
      • –swift-storage-url
      • –swift-user
      • –swift-key
      • –swift-container
      • –parallel=N
      • –cacert
      • –insecure
      • Swift authentication options
      • –swift-auth-version
      • –swift-tenant
      • –swift-tenant-id
      • –swift-region
      • –swift-password
      • –swift-user-id
      • –swift-project
      • –swift-project-id
      • –swift-domain
      • –swift-domain-id

    The xbcloud Binary¶

    The purpose of xbcloud is to download and upload full or part of xbstream archive from/to the cloud. xbcloud will not overwrite the backup with the same name. xbcloud accepts input via a pipe from xbstream so that it can be invoked as a pipeline with xtrabackup to stream directly to the cloud without needing a local storage.

    Note

    In a Bash shell, the $? parameter returns the exit code from the last binary. If you use pipes the ${PIPESTATUS[x]} array parameter returns the exit codes for each binary in the pipe string.

    $ xtrabackup --backup --stream=xbstream --target-dir=/storage/backups/ | xbcloud put [options] full_backup
    
    > true | false
    > echo $?
    > 1
    
    # with PIPESTATUS
    > true | false
    > echo ${PIPESTATUS[0]} ${PIPESTATUS[1]}
    > 0 1
    

    The xbcloud binary stores each chunk as a separate object with a name backup_name/database/table.ibd.NNNNNNNNNNNNNNNNNNNN, where NNN... is a 0-padded serial number of chunk within a file. Size of chunk produced by xtrabackup and xbstream changed to 10MB.

    Note

    Use --read-buffer-size to adjust the chunk size.

    If you use encryption, specify both the --read-buffer-size and --encrypt-chunk-size options to adjust the chunk size.

    xbcloud has three essential operations: put, get, and delete. With these operations, backups are created, stored, retrieved, restored, and deleted. xbcloud operations clearly map to similar operations within the AWS S3 API.

    Version specific information¶

    • 2.4.25 - Added the support for Microsoft Azure Cloud Storage

    • 2.4.21 - Added s3-storage-class and google-storage-class

    • 2.4.14 - Added the support of Amazon S3, MinIO and Google Cloud Storage storage types.

    • 2.3.1-beta1 - Implemented ability to store xbcloud parameters in a .cnf file

    • 2.3.1-beta1 - Implemented support different authentication options for Swift

    • 2.3.1-beta1 - Implemented support for partial download of the cloud backups

    • 2.3.1-beta1 - xbcloud --swift-url option has been renamed to xbcloud --swift-auth-url

    • 2.3.0-alpha1 - Initial implementation

    Supported Cloud Storage Types¶

    Swift was the only option for storing backups in cloud storage until Percona XtraBackup 2.4.14.

    The xbcloud binary supports Amazon S3, Azure, MinIO and Google Cloud Storage. Amazon S3-compatible cloud storage types, such as Wasabi and Digital Ocean Spaces, are also supported. The xbcloud binary should work with other S3-compatible storage options.

    See also

    OpenStack Object Storage (“Swift”)

    Amazon Simple Storage Service

    Azure Cloud Storage

    MinIO

    Google Cloud Storage

    Wasabi

    Digital Ocean Spaces

    Usage¶

    $ xtrabackup --backup --stream=xbstream --target-dir=/tmp | xbcloud \
    put [options] <name>
    

    Creating a full backup with Swift¶

    The following example shows how to make a full backup and upload it to Swift.

    $ xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
    xbcloud put --storage=swift \
    --swift-container=test \
    --swift-user=test:tester \
    --swift-auth-url=http://192.168.8.80:8080/ \
    --swift-key=testing \
    --parallel=10 \
    full_backup
    

    Creating a full backup with Amazon S3¶

    $ xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
    xbcloud put --storage=s3 \
    --s3-endpoint='s3.amazonaws.com' \
    --s3-access-key='YOUR-ACCESSKEYID' \
    --s3-secret-key='YOUR-SECRETACCESSKEY' \
    --s3-bucket='mysql_backups'
    --parallel=10 \
    $(date -I)-full_backup
    

    The following options are available when using Amazon S3:

    Option Details
    –s3-access-key Use to supply the AWS access key ID
    –s3-secret-key Use to supply the AWS secret access key
    –s3-bucket Use supply the AWS bucket name
    –s3-region Use to specify the AWS region. The default value is us-east-1
    –s3-api-version = <AUTO|2|4> Select the signing algorithm. The default value is AUTO. In this case, xbcloud will probe.
    –s3-bucket-lookup = <AUTO|PATH|DNS> Specify whether to use bucket.endpoint.com or endpoint.com/bucket* style requests. The default value is AUTO. In this case, xbcloud will probe.
    –s3-storage-class=<name> Specify the S3 storage class. The name options are the following:
    STANDARD
    STANDARD_IA
    GLACIER

    NOTE: If you use the GLACIER storage class, the object must be restored to S3 before restoring the backup.
    Also, supports using custom S3 implementations such as MinIO or CephRadosGW.

    Creating a full backup with MinIO¶

    $ xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
    xbcloud put --storage=s3 \
    --s3-endpoint='play.minio.io:9000' \
    --s3-access-key='YOUR-ACCESSKEYID' \
    --s3-secret-key='YOUR-SECRETACCESSKEY' \
    --s3-bucket='mysql_backups'
    --parallel=10 \
    $(date -I)-full_backup
    

    Creating a full backup with Google Cloud Storage¶

    The support for Google Cloud Storage is implemented using the interoperability mode. This mode was especially designed to interact with cloud services compatible with Amazon S3.

    See also

    Cloud Storage Interoperability https://cloud.google.com/storage/docs/interoperability

    $ xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
    xbcloud put --storage=google \
    --google-endpoint=`storage.googleapis.com` \
    --google-access-key='YOUR-ACCESSKEYID' \
    --google-secret-key='YOUR-SECRETACCESSKEY' \
    --google-bucket='mysql_backups'
    --parallel=10 \
    $(date -I)-full_backup
    

    The following options are available when using Google Cloud Storage:

    • –google-access-key = <ACCESS KEY ID>

    • –google-secret-key = <SECRET ACCESS KEY>

    • –google-bucket = <BUCKET NAME>

    • –google-storage-class=name

    Note

    The Google storage class name options are the following:

    • STANDARD

    • NEARLINE

    • COLDLINE

    • ARCHIVE

    See also: Google storage classes

    Supplying parameters¶

    Each storage type has mandatory parameters that you can supply on the command line, in a configuration file, and via environment variables.

    Configuration files¶

    The parameters the values of which do not change frequently can be stored in my.cnf or in a custom configuration file. The following example is a template of configuration options under the [xbcloud] group:

    [xbcloud]
    storage=s3
    s3-endpoint=http://localhost:9000/
    s3-access-key=minio
    s3-secret-key=minio123
    s3-bucket=backupsx
    s3-bucket-lookup=path
    s3-api-version=4
    

    Note

    If you explicitly use a parameter on the command line and in a configuration file, xbcloud uses the the value provided on the command line.

    Environment variables¶

    The following environment variables are recognized. xbcloud maps them automatically to corresponding parameters applicable to the selected storage.

    • AWS_ACCESS_KEY_ID (or ACCESS_KEY_ID)

    • AWS_SECRET_ACCESS_KEY (or SECRET_ACCESS_KEY)

    • AWS_DEFAULT_REGION (or DEFAULT_REGION)

    • AWS_ENDPOINT (or ENDPOINT)

    • AWS_CA_BUNDLE

    Note

    If you explicitly use a parameter on the command line, in a configuration file, and the corresponding environment variable contains a value, xbcloud uses the the value provided on the command line or in the configuration file.

    OpenStack environment variables are also recognized and mapped automatically to corresponding swift parameters (--storage=swift).

    • OS_AUTH_URL
    • OS_TENANT_NAME
    • OS_TENANT_ID
    • OS_USERNAME
    • OS_PASSWORD
    • OS_USER_DOMAIN
    • OS_USER_DOMAIN_ID
    • OS_PROJECT_DOMAIN
    • OS_PROJECT_DOMAIN_ID
    • OS_REGION_NAME
    • OS_STORAGE_URL
    • OS_CACERT

    Shortcuts¶

    For all operations (put, get, and delete), you can use a shortcut to specify the storage type, bucket name, and backup name as one parameter instead of using three distinct parameters (–storage, –s3-bucket, and backup name per se).

    Using a shortcut syntax to provide a storage type, bucket, and backup name

    Use the following format: storage-type://bucket-name/backup-name

    $ xbcloud get s3://operator-testing/bak22 ...
    

    In this example, s3 refers to a storage type, operator-testing is a bucket name, and bak22 is the backup name. This shortcut expands as follows:

    $ xbcloud get --storage=s3 --s3-bucket=operator-testing bak22 ...
    

    You can supply the mandatory parameters not only on the command line. You may use configuration files and environment variables.

    Additional parameters¶

    xbcloud accepts additional parameters that you can use with any storage type. The --md5 parameter computes the MD5 hash value of the backup chunks. The result is stored in files that following the backup_name.md5 pattern.

    $ xtrabackup --backup --stream=xbstream \
    --parallel=8 2>backup.log | xbcloud put s3://operator-testing/bak22 \
    --parallel=8 --md5 2>upload.log
    

    You may use the --header parameter to pass an additional HTTP header with the server side encryption while specifying a customer key.

    Example of using –header for AES256 encryption

    $ xtrabackup --backup --stream=xbstream --parallel=4 | \
    xbcloud put s3://operator-testing/bak-enc/ \
    --header="X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \
    --header="X-Amz-Server-Side-Encryption-Customer-Key: CuStoMerKey=" \
    --header="X-Amz-Server-Side-Encryption-Customer-Key-MD5: CuStoMerKeyMd5==" \
    --parallel=8
    

    The --header parameter is also useful to set the access control list (ACL) permissions: --header="x-amz-acl: bucket-owner-full-control

    Restoring with Swift¶

    xbcloud get [options] <name> [<list-of-files>] | xbstream -x
    

    The following example shows how to fetch and restore the backup from Swift:

    $ xbcloud get --storage=swift \
    --swift-container=test \
    --swift-user=test:tester \
    --swift-auth-url=http://192.168.8.80:8080/ \
    --swift-key=testing \
    full_backup | xbstream -xv -C /tmp/downloaded_full
    
    $ xtrabackup --prepare --target-dir=/tmp/downloaded_full
    $ xtrabackup --copy-back --target-dir=/tmp/downloaded_full
    

    Restoring with Amazon S3¶

    $ xbcloud get s3://operator-testing/bak22 \
    --s3-endpoint=https://storage.googleapis.com/ \
    --parallel=10 2>download.log | xbstream -x -C restore --parallel=8
    

    Incremental backups¶

    First, make the full backup which is the base for an incremental backup:

    $ xtrabackup --backup --stream=xbstream --extra-lsndir=/storage/backups/ \
    --target-dir=/storage/backups/ | xbcloud put \
    --storage=swift --swift-container=test_backup \
    --swift-auth-version=2.0 --swift-user=admin \
    --swift-tenant=admin --swift-password=xoxoxoxo \
    --swift-auth-url=http://127.0.0.1:35357/ --parallel=10 \
    full_backup
    

    Then make the incremental backup:

    $ xtrabackup --backup --incremental-basedir=/storage/backups \
    --stream=xbstream --target-dir=/storage/inc_backup | xbcloud put \
    --storage=swift --swift-container=test_backup \
    --swift-auth-version=2.0 --swift-user=admin \
    --swift-tenant=admin --swift-password=xoxoxoxo \
    --swift-auth-url=http://127.0.0.1:35357/ --parallel=10 \
    inc_backup
    

    Preparing an incremental backup¶

    To prepare a backup, download the full backup:

    $ xbcloud get --swift-container=test_backup \
    --swift-auth-version=2.0 --swift-user=admin \
    --swift-tenant=admin --swift-password=xoxoxoxo \
    --swift-auth-url=http://127.0.0.1:35357/ --parallel=10 \
    full_backup | xbstream -xv -C /storage/downloaded_full
    

    Prepare the downloaded full backup:

    $ xtrabackup --prepare --apply-log-only --target-dir=/storage/downloaded_full
    

    After the full backup has been prepared, download the incremental backup:

    $ xbcloud get --swift-container=test_backup \
    --swift-auth-version=2.0 --swift-user=admin \
    --swift-tenant=admin --swift-password=xoxoxoxo \
    --swift-auth-url=http://127.0.0.1:35357/ --parallel=10 \
    inc_backup | xbstream -xv -C /storage/downloaded_inc
    

    Prepare the incremental backup:

    $ xtrabackup --prepare --apply-log-only \
    --target-dir=/storage/downloaded_full \
    --incremental-dir=/storage/downloaded_inc
    
    $ xtrabackup --prepare --target-dir=/storage/downloaded_full
    

    Partial download of the cloud backup¶

    If you don’t want to download the entire backup to restore a database you can restore only specific tables:

    $ xbcloud get --swift-container=test_backup
    --swift-auth-version=2.0 --swift-user=admin \
    --swift-tenant=admin --swift-password=xoxoxoxo \
    --swift-auth-url=http://127.0.0.1:35357/ full_backup \
    ibdata1 sakila/payment.ibd \
    > /storage/partial/partial.xbs
    
    $ xbstream -xv -C /storage/partial < /storage/partial/partial.xbs
    

    This command downloads the ibdata1 table and the sakila/payment.ibd table from a full backup.

    Command-line options¶

    xbcloud has the following command line options:

    –storage=[swiftAmazon S3google]¶

    Cloud storage option. xbcloud supports Swift, MinIO, and AWS S3. The default value is swift.

    –swift-auth-url¶

    URL of Swift cluster.

    –swift-url¶

    Renamed to xbcloud –swift-auth-url

    –swift-storage-url¶

    xbcloud attempts to get object-store URL for a specified region (if any specified) from the keystone response. One can override that URL by passing –swift-storage-url=URL argument.

    –swift-user¶

    Swift username (X-Auth-User, specific to Swift)

    –swift-key¶

    Swift key/password (X-Auth-Key, specific to Swift)

    –swift-container¶

    Container to backup into (specific to Swift)

    –parallel=N¶

    Maximum number of concurrent upload/download requests. Default is 1.

    –cacert¶

    Path to the file with CA certificates

    –insecure¶

    Do not verify servers certificate

    Swift authentication options¶

    Swift specification describe several authentication options. xbcloud can authenticate against keystone with API version 2 and 3.

    –swift-auth-version¶

    Specifies the swift authentication version. Possible values are: 1.0 - TempAuth, 2.0 - Keystone v2.0, and 3 - Keystone v3. Default value is 1.0.

    For v2 additional options are:

    –swift-tenant¶

    Swift tenant name.

    –swift-tenant-id¶

    Swift tenant ID.

    –swift-region¶

    Swift endpoint region.

    –swift-password¶

    Swift password for the user.

    For v3 additional options are:

    –swift-user-id¶

    Swift user ID.

    –swift-project¶

    Swift project name.

    –swift-project-id¶

    Swift project ID.

    –swift-domain¶

    Swift domain name.

    –swift-domain-id¶

    Swift domain ID.

    Contact us

    For free technical help, visit the Percona Community Forum.

    To report bugs or submit feature requests, open a JIRA ticket.

    For paid support and managed or consulting services , contact Percona Sales.

    2023-02-27
    Percona LLC and/or its affiliates, © 2024 Cookie Preferences
    Made with Material for MkDocs