Remote backup storage¶
On this page:
Overview¶
Percona Backup for MongoDB supports the following types of remote backup storage:
S3 compatible storage
Percona Backup for MongoDB should work with other S3-compatible storages, but was only tested with the following ones:
As of version 1.3.2, Percona Backup for MongoDB supports server-side encryption for S3 buckets with customer managed keys stored in AWS KMS.
New in version 1.7.0: You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the pbm logs
output so that you can debug and diagnose S3 request issues or failures.
To enable S3 debug logging, set the storage.s3.DebugLogLevel
option in Percona Backup for MongoDB configuration. The supported values are: LogDebug
, Signing
, HTTPBody
, RequestRetries
, RequestErrors
, EventStreamBody
.
Starting with version 1.7.0, Percona Backup for MongoDB supports Amazon S3 storage classes. Knowing your data access patterns, you can set the S3 storage class in Percona Backup for MongoDB configuration. When Percona Backup for MongoDB uploads data to S3, the data is distributed to the corresponding storage class. The support of S3 bucket storage types allows you to effectively manage S3 storage space and costs.
To set the storage class, specify the storage.s3.storageClass
option in Percona Backup for MongoDB configuration file
storage:
type: s3
s3:
storageClass: INTELLIGENT_TIERING
When the option is undefined, the S3 Standard storage type is used.
See also
As of version 1.7.0, you can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the options storage.s3.retryer.numMaxRetries
, storage.s3.retryer.minRetryDelay
and storage.s3.retryer.maxRetryDelay
in Percona Backup for MongoDB configuration.
retryer:
numMaxRetries: 3
minRetryDelay: 30
maxRetryDelay: 5
This upload retry increases the chances of data upload completion in cases of unstable connection.
New in version 1.7.0: Percona Backup for MongoDB supports data upload to S3-like storage that supports self-issued TLS certificates. To make this happen, disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration:
$ pbm config --set storage.s3.insecureSkipTLSVerify=True
Warning
Use this option with caution as it might leave a hole for man-in-the-middle attacks.
Remote Filesystem Server Storage
This storage must be a remote file server mounted to a local directory. It is the responsibility of the server administrators to guarantee that the same remote directory is mounted at exactly the same local path on all servers in the MongoDB cluster or non-sharded replica set.
Warning
Percona Backup for MongoDB uses the directory as if it were any normal directory, and does not attempt to confirm it is mounted from a remote server. If the path is accidentally a normal local directory, errors will eventually occur, most likely during a restore attempt. This will happen because pbm-agent processes of other nodes in the same replica set can’t access backup archive files in a normal local directory on another server.
Local Filesystem Storage
This cannot be used except if you have a single-node replica set. (See the warning note above as to why). We recommend using any object store you might be already familiar with for testing. If you don’t have an object store yet, we recommend using MinIO for testing as it has simple setup. If you plan to use a remote filesytem-type backup server, please see the Remote Filesystem Server Storage above.
Microsoft Azure Blob Storage
As of v1.5.0, you can use Microsoft Azure Blob Storage as the remote backup storage for Percona Backup for MongoDB.
This gives users a vendor choice. Companies with Microsoft-based infrastructure can set up Percona Backup for MongoDB with less administrative efforts.
Note
Regardless of the remote backup storage you use, grant the List/Get/Put/Delete
permissions to this storage for the user identified by the access credentials.
The following example shows the permissions configuration to the pbm-testing
bucket on the AWS S3 storage.
{
"Version": "2021-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::pbm-testing"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::pbm-testing/*"
}
]
}
Please refer to the documentation of your selected storage for the data access management.
See also
AWS documentation: Controlling access to a bucket with user policies
Google Cloud Storage documentation: Overview of access control
Microsoft Azure documentation: Assign an Azure role for access to blob data
MinIO documentation : Policy Management
Example config files¶
Provide the remote backup storage configuration as a YAML config file. The following are the examples of config fles for supported remote storages. For how to insert the config file, see Insert the whole Percona Backup for MongoDB config from a YAML file.
S3-compatible remote storage
Amazon Simple Storage Service
storage:
type: s3
s3:
region: us-west-2
bucket: pbm-test-bucket
prefix: data/pbm/backup
credentials:
access-key-id: <your-access-key-id-here>
secret-access-key: <your-secret-key-here>
serverSideEncryption:
sseAlgorithm: aws:kms
kmsKeyID: <your-kms-key-here>
GCS
storage:
type: s3
s3:
region: us-east1
bucket: pbm-testing
prefix: pbm/test
endpointUrl: https://storage.googleapis.com
credentials:
access-key-id: <your-access-key-id-here>
secret-access-key: <your-secret-key-here>
MinIO
storage:
type: s3
s3:
endpointUrl: "http://localhost:9000"
region: my-region
bucket: pbm-example
prefix: data/pbm/test
credentials:
access-key-id: <your-access-key-id-here>
secret-access-key: <your-secret-key-here>
Remote filesystem server storage
storage:
type: filesystem
filesystem:
path: /data/local_backups
Microsoft Azure Blob Storage
storage:
type: azure
azure:
account: <your-account>
container: <your-container>
prefix: pbm
credentials:
key: <your-access-key>
For the description of configuration options, see Configuration file options.