Configuring Secure Clusters for Cross-Cluster Mirroring and Replication

Describes configuring clusters for cross-cluster operations such as mirroring and replication.

Cross-cluster tickets are required on secure clusters that need to pull data from another secure cluster and on secure clusters that need to push data to another secure cluster. For example:
  • Volume mirroring is a pull operation. The volume data is pulled by the destination cluster from the source cluster. Since the destination cluster performs the operation, the destination cluster receives a ticket that is generated on the source cluster.
  • Table and streams replication is a push operation. The table or stream data is pushed from the source cluster to the destination cluster. Since the source cluster performs the operation, the source cluster receives a ticket that is generated on the destination cluster.

You can configure secure clusters for cross-cluster mirroring and replication manually (as described in Setting up Secure Clusters Manually for Cross-Cluster Mirroring and Setting up Secure Clusters Manually for Cross-Cluster Replication). You can configure secure clusters automatically, by running the configure-crosscluster.sh utility. This utility configures the clusters for both mirroring and replication in both directions. For more information, see configure-crosscluster.sh.

Parent topic: Setting Up Cross-Cluster Security

Setting up Secure Clusters Manually for Cross-Cluster Mirroring

To set up secure clusters for cross-cluster mirroring:
  1. Verify that the user for whom you are configuring access, exists in the registry on both the clusters and has the following permissions:
    • Permissions to create volumes on the source cluster.
    • Permissions to mirror volumes on the destination cluster.
    You can set up access for the mapr user, who already has permissions to create volumes and mirror volumes.
  2. Configure clusterA for communicating with the other clusters by editing mapr-clusters.conf file on each node clusterA to specify the hostname or IP address of the CLDB nodes on the other clusters.
    For example, suppose:
    • clusterA’s /opt/mapr/conf/mapr-clusters.conf file contains the following: 
      clusterA.cluster.com secure=true perfnode50.lab:7222
    • clusterB’s /opt/mapr/conf/mapr-clusters.conf file contains the following: 
      clusterB.cluster.com secure=true perfnode100.lab:7222

    Perform the following steps to configure the nodes on the clusters:

    1. On any node in clusterA, append the first entry from clusterB’s mapr-clusters.conf file, entry which is prefixed with the cluster name, to the end of clusterA’s mapr-clusters.conf file.
      Note that clusterA’s entry must be the first line of the mapr-clusters.conf file: 
      clusterA.cluster.com secure=true perfnode50.lab:7222
clusterB.cluster.com secure=true perfnode100.lab:7222
      The clusterA’s mapr-clusters.conf file now contains two entries.
    2. Copy the updated /opt/mapr/conf/mapr-clusters.conf file to all the other nodes in clusterA.
    3. On any node in clusterB, append the first entry from clusterA’s mapr-clusters.conf file, entry which is prefixed with the cluster name, to the end of the remote cluster’s mapr-clusters.conf file.
      Note that clusterB’s entry must be the first line of mapr-clusters.conf file: 
      clusterB.cluster.com secure=true perfnode100.lab:7222
clusterA.cluster.com secure=true perfnode50.lab:7222
      The clusterB’s mapr-clusters.conf file now contains two entries.
    4. Copy the updated /opt/mapr/conf/mapr-clusters.conf file to all the nodes in clusterB.
    See mapr-clusters.conf.
  3. Log in to any node on the source cluster (ClusterA) and perform the following steps:
    1. Generate a cross-cluster ticket for the destination cluster for this user.
      For example, to generate a cross-cluster for destination cluster, run the following command on the source cluster: 
      source$ /opt/mapr/bin/maprlogin generateticket -type crosscluster -out /tmp/crossclusterticket -user destinationclusteruser
    2. Copy the cross-cluster ticket file to any node on the destination cluster (clusterB).
      For example: 
      source$ scp /tmp/crossclusterticket mapr@<dest-ip>:/tmp/sourceClusterTicketFile
  4. Log in to the node on the destination cluster (clusterB) where the cross-cluster ticket was copied, and perform the following steps:
    1. Merge the cross-cluster ticket file with the /opt/mapr/conf/maprserverticket file on the node.
      For example, to merge, run the following command: 
      dest$ cat /tmp/sourceClusterTicketFile >> /opt/mapr/conf/maprserverticket
    2. Copy the /opt/mapr/conf/maprserverticket file to the CLDB nodes on the destination cluster.
  5. Perform the following step on clusterB to ensure that the ssl_truststore file has signers for all the clusters. Merge the ssl_truststore files with the /opt/mapr/server/manageSSLKeys.sh tool.

    Use one of the following three ways to merge tickets:

    1. Make a copy of the source ssl_truststore. Create a file with the password of the source truststore. Finally, merge the source and the destination truststores.

      1. Copy the ssl_truststore file from cluster A: 
        scp mapr@<remote-ip>:/opt/mapr/conf/ssl_truststore /tmp/clusterA_ssl_truststore
      2. Create a file with password of the source trustore (can be obtained from the ssl-client.xml file of clusterA) as the content. Assume you copied it to file /tmp/stpassword.
      3. Merge on cluster B: 
        manageSSLKeys.sh merge /tmp/clusterA_ssl_truststore /opt/mapr/conf/ssl_truststore /tmp/stpassword

    2. Use the copytruststore option of manageSSLKeys.sh and create a copy of the truststore. Then copy it to the destination node using SCP, and finally run merge without any additional options. For example:

      1. Create new ssl_truststore using the copytruststore option on cluster A: 
        manageSSLKeys.sh copytruststore /tmp/clusterA_ssl_truststore
      2. Copy the ssl_truststore from cluster A: 
        scp mapr@<remote-ip>:/opt/mapr/conf/clusterA_ssl_truststore /tmp/clusterA_ssl_truststore
      3. Merge on cluster B: 
        manageSSLKeys.sh merge /tmp/clusterA_ssl_truststore /opt/mapr/conf/ssl_truststore
    3. Use the copytruststore option of manageSSLKeys.sh and create a custom password for the copy of truststore. Copy this truststore to the destination node using SCP. Then create a file with password of the copied truststore as the content, and pass the filename as the argument to merge. For example:
      1. Create a custom password for the copy of truststore: 
        manageSSLKeys.sh copytruststore /tmp/ssl_truststore3 abcdef
      2. Copy this password to a file: 
        echo abcdef >/tmp/a
      3. Copy the ssl_truststore and password from cluster A to cluster B: 
        scp mapr@<remote-ip>:/tmp/ssl_truststore3 /tmp/ssl_truststore3
        scp mapr@<remote-ip>:/tmp/a  /tmp/a
      4. Merge on cluster B: 
        manageSSLKeys.sh merge /tmp/ssl_truststore3 /opt/mapr/conf/ssl_truststore /tmp/a
  6. Copy the merged ssl_truststore file to every node on clusterB.
  7. Optional: If your clusters are secure, configure your source cluster so that you can use the Control System to set up and administer table replication from the source to the destination cluster.
    These steps make it convenient to use the Control System for setting up and managing replication involving two secure clusters. However, before following them, perform these prerequisite tasks.
    Note:
    • Ensure that both clusters are managed by the same team or group. The UIDs and GIDs of the users that are able to log in to the Control System on the source cluster must exactly match their UIDs and GIDs on the destination cluster. This restriction applies only to access to both clusters through the Control System, and does not apply to access to both clusters through the maprcli. If the clusters are managed by different teams or groups, use the maprcli instead of the Control System to set up and manage table replication involving two secure clusters.
    • Ensure that the proper file-system and table permissions are in place on both clusters. Otherwise, any user who can log into the Control System and has the same UID or GID on the destination cluster will be able to set up replication either from the source cluster to the destination cluster or vice versa. A user could create one or more tables on the destination cluster, enable replication to them from the source cluster, load the new tables with data from the source cluster, and start replication. A user could also create tables on the source cluster, enable replication to them from tables in the destination cluster, load the new tables with data from the destination cluster, and start replication.
    1. On the source cluster, generate a service ticket by using the maprlogin command: 
      maprlogin generateticket -type service -cluster <destination cluster>
-user mapr -duration <duration> -out <output folder>

      Where -duration is the length of time before the ticket expires. You can specify the value in either of these formats:

      • [Days:]Hours:Minutes
      • Seconds
    2. To every node of the source cluster, add the service ticket to the file /opt/mapr/conf/mapruserticket file that was created when you secured the source cluster: 
      at <path and filename of the service ticket> >> /opt/mapr/conf/mapruserticket
    3. Restart the web server by running the maprcli node services command. For the syntax of this command, see node services.
    4. Add the following two properties to the core-site.xml file. For Hadoop 2.7.0, edit the file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml. 
      <property>
      <name>hadoop.proxyuser.mapr.hosts</name>
      <value>*</value>
 </property>
 <property>
      <name>hadoop.proxyuser.mapr.groups</name>
      <value>*</value>
 </property>
  8. Perform the steps to verify configuration for mirroring.
You can now create mirror volumes on the destination cluster and set up a schedule to pull data from the volumes on the source cluster. However, you cannot create volumes on the source cluster that pull data from volumes in the destination cluster, because the setup described above is unidirectional. To configure the clusters for bidirectional mirroring, repeat the steps above, by switching the source and destination clusters.

For example, suppose there are two clusters, clusterA and clusterB, and you performed the steps above for clusterA as the source cluster and clusterB as the destination cluster. After you complete the steps above, your destination cluster, clusterB can pull data from volumes on clusterA. For clusterA to mirror data on clusterB, perform the steps above with clusterB as the source cluster and clusterA as the destination cluster.

Setting up Secure Clusters Manually for Cross-Cluster Replication

To set up secure clusters for cross-cluster replication:
  1. Verify that the user, for whom you are configuring access, exists in the registry on the destination cluster.
  2. Log in to any node on the destination cluster and perform the following steps:
    1. Generate a cross-cluster ticket for the source cluster.
      For example, to generate a cross-cluster for the source cluster, run the following command on the destination cluster: 
      dest$ /opt/mapr/bin/maprlogin generateticket -type crosscluster -out /tmp/crossclusterticket -user destinationclusteruser
    2. Copy the cross-cluster ticket file to any node on the source cluster.
      For example: 
      dest$ scp /tmp/crossclusterticket mapr@<source-ip>:/tmp/sourceClusterTicketFile
  3. Log in to the node in the source cluster where the cross-cluster ticket was copied, and perform the following steps:
    1. Merge the cross-cluster ticket file with the /opt/mapr/conf/maprserverticket file on the node.
      For example, to merge, run the following command: 
      cat /tmp/destinationClusterTicketFile >> /opt/mapr/conf/maprserverticket
    2. Copy the /opt/mapr/conf/maprserverticket file to all the nodes on the source cluster.
  4. Configure the Gateway for table and streams replication.
    See Configuring Gateways for Table and Stream Replication for more information.
  5. Perform the steps to verify configuration for replication.
You can now set up volumes on the source cluster to push data to replicas on the destination cluster. However, you cannot create replicas on the source cluster that get data from volumes in the destination cluster because the setup described above is unidirectional. To configure the clusters for bidirectional replication, repeat the steps above by switching the source and destination clusters.

For example, suppose there are two clusters, clusterA and clusterB, and you performed the steps above for clusterA as the source cluster and clusterB as the destination cluster. After you complete the steps above, your source cluster, clusterA can push data to replicas on clusterB. For clusterB to replicate data on clusterA, perform the steps above with clusterB as the source cluster, and clusterA as the destination cluster.

Verifying Cross-Cluster Configuration for Mirroring and Replication

You can verify the cross-cluster configuration for:
  1. Mirroring by logging in to a node on the destination cluster as the user for whom access was configured, and creating a mirror volume on the destination cluster for a volume on the source cluster.
    You can create mirror volumes using the Control System and/or the CLI.
  2. Replication by logging in to a node on the source cluster as the user for whom access was configured and creating a replica in the destination cluster for a volume, table, and stream on the source cluster.
    You can create replicas using the Control System and the CLI. To set up replication on secure clusters for: