Enabling Security

Describes how to enable security for the cluster, platform, ecosystem components, and network-based connections.

The following steps enable security for the cluster, wire-level encryption for the platform and ecosystem components, authentication for all network-based connections, and optionally, data at rest encryption on the cluster.

When you set up a cluster, run the configure.sh script on each node that you want to add to the cluster. After you enable security, review the System Behavior Changes After Enabling Security.

Parent topic: Step 6: Configure Nodes

Basic Procedure

To enable security for the cluster, follow these steps in order:

  1. If the cluster is running, shut it down.
  2. Delete the ssl_keystore, ssl_truststore, ssl_truststore.p12, ssl_truststore.pem, ssl-server.xml, ssl-client.xml, ssl_usertruststore, and ssl_userkeystore files on a machine where wire-level security is not enabled, as the configure.sh script fails if you already have these files in the directory.
    Note: The ssl_keystore, ssl_keystore.p12, ssl_keystore.pem, ssl_truststore, ssl_truststore.p12, ssl_truststore.pem, ssl-server.xml, and ssl-client.xml files are generated during installation of the Web server even if you did not enable security.
    For example, to delete the files, run the following commands:
    cd /opt/mapr/conf 
/bin/rm -f ssl_keystore ssl_keystore.p12 ssl_keystore.pem 
    ssl_truststore ssl_truststore.p12 ssl_truststore.pem ssl-server.xml ssl-client.xml
    ssl_usertruststore ssl_userkeystore
    If you are re-running the script due to an invocation error from a previous run, the cldb.key and maprserverticket files may have already been created. Delete these files, as the script fails if you already have these files in the directory. For example, run the following command to delete these files:
    cd /opt/mapr/conf
/bin/rm -f cldb.key maprserverticket ssl_keystore ssl_keystore.p12 ssl_keystore.pem 
   ssl_truststore ssl_truststore.p12 ssl_truststore.pem ssl-server.xml ssl-client.xml
   ssl_usertruststore ssl_userkeystore
  3. Run the configure.sh script with the -secure -genkeys -dare options on the first CLDB node in your cluster: 
    /opt/mapr/server/configure.sh -secure -dare -genkeys -Z <Zookeeper_node_list> -C <CLDB_node_list> -N <cluster_name>
    where both <Zookeeper_node_list> and <CLDB_node_list> have the form hostname[:port_no][,hostname[:port_no]...] and -N <cluster_name> specifies the cluster name. For the hostname, specify a fully-qualified domain name (FQDN) as described in Connectivity. Do not specify an alias or IP address. The -dare option is required only if you wish to enable data at rest encryption at the cluster-level.
    Important: You must run configure.sh -dare -genkeys only once on one CLDB node, since the resulting files must be copied to other nodes.
    This command generates the following files in the /opt/mapr/conf directory:
    • cldb.key
    • dare.master.key
    • maprserverticket
    • ssl_keystore
    • ssl_keystore.p12
    • ssl_keystore.pem
    • ssl_keystore-signed.pem
    • ssl_truststore
    • ssl_truststore.p12
    • ssl_truststore.pem
    • ssl-server.xml
    • ssl-client.xml
    • ssl_userkeystore
    • ssl_userkeystore.csr
    • ssl_userkeystore.p12
    • ssl_userkeystore.pem
    • ssl_userkeystore-signed.pem
    • ssl_usertruststore
    • ssl_usertruststore.p12
    • ssl_usertruststore.pem
    The command generates the following files in the opt/mapr/conf/ca directory:
    • chain-ca.pem
    • root-ca.pem
    • signing-ca.pem
    Note: The dare.master.key file is generated only if data-at-rest encryption is enabled on the cluster.
    Tip: A comprehensive listing of the Trust and Key Store files is at: Understanding the Truststore and Keyfiles.
  4. Copy the cldb.key to any node with the CLDB or Zookeeper service installed, and copy the dare.master.key to any node with the CLDB service installed.
    Note: Copy the dare.master.key file only if you are enabling data at rest encryption on the cluster.
  5. Verify ownership and permissions:
    1. Verify that the following files in the /opt/mapr/conf directory are owned by the user that runs cluster services. This user is mapr by default:
      • ssl_keystore
      • ssl_keystore.csr
      • ssl_keystore.p12
      • ssl_keystore.pem
      • ssl_keystore-signed.pem
      • ssl_truststore
      • ssl_truststore.p12
      • ssl_truststore.pem
      • ssl-server.xml
      • ssl-client.xml
      • ssl_userkeystore
      • ssl_userkeystore.csr
      • ssl_userkeystore.p12
      • ssl_userkeystore.pem
      • ssl_userkeystore-signed.pem
      • ssl_usertruststore
      • ssl_usertruststore.p12
      • ssl_usertruststore.pem
    2. Verify that the maprserverticket has its permission-mode bits set to 600
    3. Verify that the following files have their UNIX permission-mode bits set to 400:
      • ssl_keystore
      • ssl_keystore.csr
      • ssl_keystore.p12
      • ssl_keystore.pem
      • ssl_keystore-signed.pem
      • ssl_userkeystore
      • ssl_userkeystore.csr
      • ssl_userkeystore.p12
      • ssl_userkeystore.pem
      • ssl_userkeystore-signed.pem
    4. Verify that the following files are readable to all users:
      • ssl_truststore
      • ssl_truststore.p12
      • ssl_truststore.pem
      • ssl_usertruststore
      • ssl_usertruststore.p12
      • ssl_usertruststore.pem
  6. Copy the following files to the /opt/mapr/conf directory of every node in the cluster:
    • maprserverticket
    • ssl_keystore
    • ssl_keystore.csr
    • ssl_keystore.p12
    • ssl_keystore.pem
    • ssl_keystore-signed.pem
    • ssl_truststore
    • ssl_truststore.p12
    • ssl_truststore.pem
    • ssl-server.xml
    • ssl-client.xml
    • ssl_userkeystore
    • ssl_userkeystore.csr
    • ssl_userkeystore.p12
    • ssl_userkeystore.pem
    • ssl_userkeystore-signed.pem
    • ssl_usertruststore
    • ssl_usertruststore.p12
    • ssl_usertruststore.pem
    Copy the following files to the /opt/mapr/conf/ca directory of every node in the cluster. Create the /opt/mapr/conf/ca directory if it does not exist:
    • chain-ca.pem
    • root-ca.pem
    • signing-ca.pem
  7. Run configure.sh on each existing node in the cluster using the same arguments as in Step 3 but without the -genkeys option. 
    /opt/mapr/server/configure.sh -secure -dare -Z <Zookeeper_node_list> -C <CLDB_node_list> -N <cluster_name>
    The -secure option indicates that security must be enabled on the node where the command is run and the -dare option indicates that data at rest encryption must be enabled on the node and must be specified only if it was specified in Step 3.
    Important: You must also do this on any nodes that you add to the cluster in the future.
  8. Copy the ssl_truststore, ssl_truststore.p12, ssl_truststore.pem and ssl-client.xml files to any client nodes outside the cluster.
    Important: If you run configure.sh -secure on a node before you copy the necessary files to that node, the command fails.
  9. When using KMIP, after rekeying, copy the contents of /opt/mapr/conf/tokens to all the CLDB nodes in the cluster.
  10. Optionally, enable encrypted quorum ZooKeeper communication. See zoo.cfg for more information.
  11. Log in as the mapr superuser using the maprlogin command: maprlogin password (in this command, password is literal text).
  12. If clients will connect to multiple secure clusters, merge the ssl_truststore files with the /opt/mapr/server/manageSSLKeys.sh tool. See Setting Up the Client for more information on data-fabric clients.

    Use one of the following three ways to merge tickets:

    1. Create a file with password of sourceTrustore (can be obtained from the ssl-client.xml file of the source cluster) as the content, and pass the filename as the argument to merge: 
      manageSSLKeys.sh merge <in trust store> <out trust store> [inPasswordFile (needed to override default password)]
    2. Use the copytruststore option of manageSSLKeys.sh and create a copy of truststore. Then copy it to the destination node using SCP, and finally run merge without any additional options. For example: 
      [root@perfnode223 ~]# ./manageSSLKeys.sh copytruststore /tmp/ssl_truststore2
[root@perfnode223 ~]# ./manageSSLKeys.sh merge /tmp/ssl_truststore2 /opt/mapr/conf/ssl_truststore
Merging certificates from /tmp/ssl_truststore2 into existing /opt/mapr/conf/ssl_truststore
    3. Use the copytruststore option of manageSSLKeys.sh and create a custom password for the copy of truststore. Copy this truststore to the destination node using SCP. Then create a file with password of the copied truststore as the content, and pass the filename as the argument to merge. For example: 
      [root@perfnode223 ~]# ./manageSSLKeys.sh copytruststore /tmp/ssl_truststore3 abcdef
[root@perfnode223 ~]# echo abcdef >/tmp/a 
[root@perfnode223 ~]# ./manageSSLKeys.sh merge /tmp/ssl_truststore3 /opt/mapr/conf/ssl_truststore /tmp/a
Merging certificates from /tmp/ssl_truststore3 into existing /opt/mapr/conf/ssl_truststore

Regenerating the Truststore Password

The truststore password is generated when you run /opt/mapr/server/configure.sh -genkeys. This password is stored in the /opt/mapr/conf/ssl-server.xml and the /opt/mapr/conf/ssl-client.xml files. For example:

<property>
  <name>ssl.server.truststore.password</name>
  <value>EfbNb2qPAW3iH4D_Foyl8jHWFPjEL8lD</value>
  <description>Optional. Default value is "".
  </description>
</property>    

<property>
  <name>ssl.client.truststore.password</name>
  <value>EfbNb2qPAW3iH4D_Foyl8jHWFPjEL8lD</value>
  <description>Optional. Default value is "".
  </description>
</property>

Data Fabric clients use this password information. If you change the password, you must copy the ssl-client.xml and ssl_truststore files from the cluster to the client. The upgrade instructions for each of the data-fabric clients include this step.

To regenerate the truststore password, run: 
/opt/mapr/server/manageSSLKeys.sh createrandompassword oldpassword

Here oldpassword is the password that is already present in the /opt/mapr/conf/ssl-server.xml and the /opt/mapr/conf/ssl-client.xml files.

The system then regenerates the password. The command returns the temporary location where the output is generated, the source files, and the destination where they are to be copied. Copy them manually as directed.

Advanced Procedure

In certain situations, you may opt for variant of the basic procedure. Such situations include, but are not limited to the following:

  • You are running the script on a host that is configured without a domain name.
  • You have a cluster where all the machines do not have the same domain name.
  • You wish to import your own custom certificates instead of the self-signed certificates generated by the configure.sh utility.

Running on Hosts with no Configured Domain Name

When used without the -certdomain argument, the configure.sh script discovers the domain name of the node on which it is being executed using the hostname -d command and then creates a 100-year self-signed certificate using the PKCS#1 v1.5 with SHA-512 hash function (SHA512withRSA) with a wildcard certificate with the common name (CN) *.<domain>. For example, if hostname -d returns the domain name mycompany.com, then the CN of the certificate is *.mycompany.com. This certificate works for all machines within the mycompany.com domain and can therefore be copied to all cluster nodes as specified in Step 5 in the Basic Procedure.

Certificate generation fails if the host that you are running the script from is configured without a domain name. To fix this, modify your machine configuration so that hostname -d returns a non-empty string and then run the configure.sh script.

Alternatively, re-run the script with the -certdomain option as shown in Step 3 of the Basic Procedure:

/opt/mapr/server/configure.sh -secure -genkeys -certdomain <domain_name> -Z <Zookeeper_node_list> -C <CLDB_node_list> [ -N <cluster_name> ]

Securing Clusters with Multiple Domain Names

Generally, all machines within a cluster should belong to the same domain. In the unusual case where you have a cluster with different machines belonging to different domains, applications that perform hostname verification can fail if you run the configure.sh script (as described in Step 3 of the Basic Procedure) to generate a single-domain wildcard certificate. In this case, you must provide your own multi-domain wildcard certificate and import the custom certificate into the keystore as described in Importing a Certificate Authority Signed (CA Signed) SSL Certificate Into a Data Fabric Cluster.

Using Custom Certificates

To import your own custom certificates into the keystore instead of using the self-signed certificates, see Importing a Certificate Authority Signed (CA Signed) SSL Certificate Into a Data Fabric Cluster.

Related reference