Splitting replica set across multiple data centers¶

Splitting the replica set of the database cluster over multiple Kubernetes clusters can be useful to get a fault-tolerant system in which all replicas are in different data centers.

The Operator cannot deploy MongoDB replicas to other data centers, but this solution can be achieved with a number of Operator deployments, equal to the size of your replica set. So, you will need at least 3 Operator instances: one Operator to control the replica set via cross-site replication, and at least two Operators to bootstrap the unmanaged clusters. Each cluster will contain replica set with only one member, and the Main site will manage instances from other sites as external nodes. All configuration of the replica set is done manually.

The solution has the following limitations to consider:

setting it up involves a number of manual operations, and the same applies to scaling such a manually configured replica,
backups are supported on the Main site only, not on the Replica sites.

Configuring the Main site¶

You will use the externally reachable URI for each of your replica set instances, manually overwiriting its default local fully-qualified domain name (FQDN) in the Custom Resouce manifest. Also you will need including all these host names into TLS certificates. So the first thing needed is the list of these externally reachable names. In the above example we will use the following ones:

r1.percona.local:443 URI for the cluster-name-rs0-0 (1^st replica set instance),
r2.percona.local:443 for the 2^nd replica set instance,
r3.percona.local:443 for the 3^rd replica set instance.

Following steps will allow you to prepare the Main site for cross-site replication, keeping in mind the multiple data centers deployment:

TLS certificates generated by the Operator are not suitable and it’s required to generate certificates manually on the Main site before creating a database cluster, with all names from replsetOverrides and externalNodes.

Use TLS ceritficates manual generation instruction to prepare TLS certificates with the host names from your prepared list.
Deploy your Main site as usual, with these manually generated certificates. Don’t forget to turn on Pods exposure on your Main cluster.
Now override hostname of the first replica in the replica set configuration by using the replsets.replsetOverrides subsection in the Custom Resource options manifest with the externally reachable endpoint from your externally reachable URI list:
```
...
unsafeFlags:
  replsetSize: true
replsets:
- name: rs0
   size: 1
   replsetOverrides:
     cluster-name-rs0-0:
       host: r1.percona.local:443
   ...
```
The unsafeFlags.replsetSize option in the above example is needed to create replica set with less than 3 instances.

The actual approach to make the URI reachable from the outside of your Kubernetes culster depends on the exposure type. It is different in case of the NodePort exposure , Load balancer of the cloud provider , etc. Operator won’t perform any validation for hostnames. It’s user’s responsibility to ensure connectivity.
Note

You can also add custom tags to the replset members, just to make their identication easier:
```
...
replsetOverrides:
  cluster-name-rs0-0:
    host: r1.percona.local:443
    tags:
      team: cloud
...
```

Configuring Replica sites¶

To configure Replica sites, you should deploy your Relica sites, repeating the following steps for each Kubernetes cluster you are adding:

Copy secrets from the Main site, rename them according to the cluster name you use on the Replica site (if needed), and apply.
- cluster1-ssl (SSL certificates for client connections),
- cluster1-ssl-internal (SSL certificates for replication),
- cluster1-secrets (user credentials),
- cluster1-mongodb-encryption-key (encryption key).
Deploy the database cluster on the Replica site. Don’t forgetting the following:
1. All Replica sites must be deployed with the unmanaged: true Custom Resource option. This will stop the Operator in the Replica cluster from touching the MongoDB replset configuration. Starting from this moment, only the Operator of the Main cluster will be able to modify it.
2. Backups must be disabled with the backup.enabaled: false Custom Resource option.
3. The updateStrategy Custom Resource option must be set to RollingUpdate or OnDelete.
4. In order to create a single-instance replica set, you will need to the unsafeFlags.replsetSize option to true as you did on the Main site.

Now add the new Replica site’s Pod to your Main site’s externalNodes subsection of the Custom Resource options manifest:

replsets:
- name: rs0
  size: 1
  replsetOverrides:
    cluster1-rs0-0:
      host: r1.percona.local:443
  externalNodes:
  - host: r2.percona.local:443
    votes: 1
    priority: 1

Get expert help¶

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services. Join K8S Squad to benefit from early access to features and “ask me anything” sessions with the Experts.

Community Forum Get a Percona Expert Join K8S Squad

Last update: 2024-11-15