Set up Percona Server for MongoDB cross-site replication¶
The cross-site replication involves configuring one MongoDB site as Main, and another MongoDB site as Replica to allow replication between them:
The Operator automates configuration of Main and Replica MongoDB sites, but the feature itself is not bound to Kubernetes. Either Main or Replica can run outside of Kubernetes, be regular MongoDB and be out of the Operators’ control.
This feature can be useful in several cases:
- simplify the migration of the MongoDB cluster to and from Kubernetes
- add remote nodes to the replica set for disaster recovery
Configuring the cross-site replication for the cluster controlled by the Operator is explained in the following subsections.
Exposing instances of the MongoDB cluster¶
You need to expose all Replica Set nodes (including Config Servers) through a dedicated service to ensure that Main and Replica can reach each other, like in a full mesh:
Note
Starting from v1.14, the operator configures the replset using local DNS
hostnames even if the replset is exposed. If you want to have IP addresses
in the replset configuration to achieve a multi-cluster deployment, you need
to set clusterServiceDNSMode
to External
.
This is done through the replsets.expose
, sharding.configsvrReplSet.expose
,
and sharding.mongos.expose
sections in the deploy/cr.yaml
configuration file
as follows.
spec:
replsets:
- rs0:
expose:
enabled: true
exposeType: LoadBalancer
...
sharding:
configsvrReplSet:
expose:
enabled: true
exposeType: LoadBalancer
...
The above example is using the LoadBalancer Kubernetes Service object, but there are other options (ClusterIP, NodePort, etc.).
Note
The above example will create a LoadBalancer per each Replica Set Pod. In most cases, this Load Balancer should be internet-facing for cross-region replication to work.
To list the endpoints assigned to Pods, list the Kubernetes Service objects by
executing kubectl get services -l "app.kubernetes.io/instance=CLUSTER_NAME"
command.
Configuring cross-site replication on Main site¶
The cluster managed by the Operator should be able to reach external nodes of
the Replica Sets. You can provide needed information in the
replsets.externalNodes
and sharding.configsvrReplset.externalNodes
subsections of the deploy/cr.yaml
configuration file. Following keys can
be set to specify each external Replica, both for its Replica Set and Config
Server instances:
- set
host
to URL or IP address of the external replset instance, - set
port
to the port number of the external node (or rely on the27017
default value),
Optionaly you can set the following additional keys:
priority
key sets the priority of the external node (2
by default for all local members of the cluster; external nodes should have lower priority to avoid unmanaged node being elected as a primary;0
adds the node as a non-voting member),votes
key sets the number of votes an external node can cast in a replica set election (0
by default, and0
for non-voting members of the cluster).
Here is an example:
spec:
unmanaged: false
replsets:
- name: rs0
externalNodes:
- host: rs0-1.percona.com
port: 27017
priority: 0
votes: 0
- host: rs0-2.percona.com
...
sharding:
configsvrReplSet:
size: 3
externalNodes:
- host: cfg-1.percona.com
port: 27017
priority: 0
votes: 0
- host: cfg-2.percona.com
...
The Main site will be ready for replication when you apply changes as usual:
$ kubectl apply -f deploy/cr.yaml
Getting the cluster secrets and certificates to be copied from Main to Replica¶
Main and Replica should have same Secrets objects (to have same users
credentials) and certificates. So you may need to copy them from Main.
Names of the corresponding objects are set in the users
, ssl
, and
sslInternal
keys of the Custom Resource secrets
subsection
(my-cluster-name-secrets
, my-cluster-name-ssl
, and
my-cluster-name-ssl-internal
by default).
If you can get Secrets from an existing cluster by executing the
kubectl get secret
command for each Secrets object you want to acquire:
$ kubectl get secret my-cluster-name-secrets -o yaml > my-cluster-secrets.yaml
Next remove the annotations
, creationTimestamp
, resourceVersion
,
selfLink
, and uid
metadata fields from the resulting file to make it
ready for the Replica.
You will need to further apply these secrets on Replica.
Configuring cross-site replication on Replica instances¶
When the Operator creates a new cluster, a lot of things are happening, such as
electing the Primary, generating certificates, and picking specific names. This
should not happen if we want the Operator to run the Replica site, so first
of all the cluster should be put into unmanaged state by setting the
unmanaged
key in the deploy/cr.yaml
configuration file to true. Also you
should set updateStrategy
key to OnDelete
and backup.enabled
to
false
, because Smart Updates and
backups are not allowed on unmanaged clusters.
Note
Setting unmanaged
to true will not only prevent the Operator from
controlling the Replica Set configuration, but it will also result in not
generating certificates and users credentials for new clusters.
Here is an example:
spec:
unmanaged: true
updateStrategy: OnDelete
replsets:
- name: rs0
size: 3
...
backup:
enabled: false
...
Main and Replica sites should have same Secrets objects, so don’t forget
to apply Secrets from your Main site. Names of the corresponding objects
are set in the users
, ssl
, and sslInternal
keys of the Custom
Resource secrets
subsection (my-cluster-name-secrets
,
my-cluster-name-ssl
, and my-cluster-name-ssl-internal
by default).
Copy your secrets from an existing cluster and apply each of them on your Replica site as follows:
$ kubectl apply -f my-cluster-secrets.yaml
The Replica site will be ready for replication when you apply changes as usual:
$ kubectl apply -f deploy/cr.yaml
Enabling multi-cluster Services¶
Kubernetes multi-cluster Services (MCS) is a cross-cluster discovery and invocation of Services. MCS-enabled Services become discoverable and accessible across clusters with a virtual IP address.
This feature allows splitting applications into multiple clusters combined in one fleet, which can be useful to separate logically standalone parts (i.e. stateful and stateless ones), or to address privacy and scalability requirements, etc.
Multi-cluster Services should be supported by the cloud provider. It is supported by Google Kubernetes Engine (GKE), and by Amazon Elastic Kubernetes Service (EKS).
Configuring your cluster for multi-cluster Services includes two parts:
- configure MCS with your cloud provider,
- make needed preparations with the Operator.
To set up MCS for a specific cloud provider you should follow official guides, for example ones from Google Kubernetes Engine (GKE), or from Amazon Elastic Kubernetes Service (EKS).
Warning
For EKS, you also need to create ClusterProperty objects prior to enabling multi-cluster services.
apiVersion: about.k8s.io/v1alpha1
kind: ClusterProperty
metadata:
name: cluster.clusterset.k8s.io
spec:
value: [Your Cluster identifier]
---
apiVersion: about.k8s.io/v1alpha1
kind: ClusterProperty
metadata:
name: clusterset.k8s.io
spec:
value: [Your ClusterSet identifier]
Check AWS MCS controller repository for more information.
Setting up the Operator for MCS results in registering Services for export to
other clusters using the ServiceExport object,
and using ServiceImport one to import external services. Set the following
options in the multiCluster
subsection of the deploy/cr.yaml
configuration
file to make it happen:
multiCluster.enabled
should be set totrue
,multiCluster.DNSSuffix
string should be equal to the cluster domain suffix for multi-cluster Services used by Kubernetes (svc.clusterset.local
by default).
The following example in the deploy/cr.yaml
configuration file is rather
straightforward:
...
multiCluster:
enabled: true
DNSSuffix: svc.clusterset.local
...
Apply changes as usual with the kubectl apply -f deploy/cr.yaml
command.
Note
If you want to enable multi-cluster services in a new cluster, we
recommended deploying the cluster first with multiCluster.enabled
set to
false
and enable it after replset is initialized. Having MCS enabled from
the start is prone to errors on replset initialization.
The initial ServiceExport creation and sync with the clusters of the fleet takes approximately five minutes. You can check the list of services for export and import with the following commands:
$ kubectl get serviceexport
Expected output
NAME AGE
my-cluster-name-cfg 22m
my-cluster-name-cfg-0 22m
my-cluster-name-cfg-1 22m
my-cluster-name-cfg-2 22m
my-cluster-name-mongos 22m
my-cluster-name-rs0 22m
my-cluster-name-rs0-0 22m
my-cluster-name-rs0-1 22m
my-cluster-name-rs0-2 22m
$ kubectl get serviceimport
Expected output
NAME TYPE IP AGE
my-cluster-name-cfg Headless 22m
my-cluster-name-cfg-0 ClusterSetIP ["10.73.200.89"] 22m
my-cluster-name-cfg-1 ClusterSetIP ["10.73.192.104"] 22m
my-cluster-name-cfg-2 ClusterSetIP ["10.73.207.254"] 22m
my-cluster-name-mongos ClusterSetIP ["10.73.196.213"] 22m
my-cluster-name-rs0 Headless 22m
my-cluster-name-rs0-0 ClusterSetIP ["10.73.206.24"] 22m
my-cluster-name-rs0-1 ClusterSetIP ["10.73.207.20"] 22m
my-cluster-name-rs0-2 ClusterSetIP ["10.73.193.92"] 22m
Note
ServiceExport objects are created automatically by the Percona Server for MongoDB Operator. ServiceImport objects, on the other hand, are not controlled by the operator. If you need to troubleshoot ServiceImport objects you must check the MCS controller installed by your cloud provider.
After ServiceExport object is created, exported Services can be resolved from
any Pod in any fleet cluster as
SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local
.
Note
This means that ServiceExports with the same name and namespace will be recognized as a single combined Service.
MCS can charge cross-site replication with additional limitations specific to
the cloud provider. For example, GKE demands all participating Pods to be in the
same project.
Also, default
Namespace should be used with caution: your cloud provider
may not allow
exporting Services from it to other clusters.
Applying MCS to an existing cluster¶
Additional actions are needed to turn on MCS for the already-existing non-MCS cluster.
-
You need to restart the Operator after editing the
multiCluster
subsection keys and applyingdeploy/cr.yaml
. Find the Operator’s Pod name in the output of thekubectl get pods
command (it will be something likepercona-server-mongodb-operator-d859b69b6-t44vk
) and delete it as follows:$ kubectl delete percona-server-mongodb-operator-d859b69b6-t44vk
-
If you are enabling MCS for a running cluster after upgrading from the Operator version
1.11.0
or below, you need rotating multi-domain (SAN) certificates. Do this by pausing the cluster and deleting TLS Secrets.