Configuring NFSv4 Server for Kerberos

Configure NFSv4 Server for Kerberos with Active Directory Server

The following procedure describes how to configure the data-fabric NFSv4 server to work with the Kerberos available with Active Directory server. Before configuring the data-fabric NFSv4 server, ensure that Active Directory server is installed and all the nodes on the cluster have joined that Active Directory server. The following procedure requires the NFSv4 server to run under user mapr and group maprgrp.

In an Active Directory server environment, join the cluster nodes to the Active Directory server.
Follow the sample procedure here or consult with your system administrator for assistance with installing and joining the nodes to Active Directory server.
Check if Kerberos tickets for host and NFS service principal are present, by running the following command:
```
# klist
klist: No credentials cache found (filename: /tmp/krb5cc_0)
```

Ensure host principal is available by checking to see if existing keys are present on the node.

For example, when you run the following command, the output should look similar to the following output for nfs4ad.com domain:

# klist -kt
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 04/10/2018 23:51:24 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:24 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:24 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:24 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:24 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:24 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:25 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:25 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:25 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   2 04/10/2018 23:51:25 host/ATSQA4-161@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM
   2 04/10/2018 23:51:25 ATSQA4-161$@NFS4AD.COM

Generate the host ticket by running the kinit command.

For example:

[root@atsqa4-161 ~]# kinit -k ATSQA4-161$
[root@atsqa4-161 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ATSQA4-161$@NFS4AD.COM
Valid starting       Expires              Service principal
04/11/2018 03:04:38  04/11/2018 13:04:38  krbtgt/NFS4AD.COM@NFS4AD.COM
        renew until 04/18/2018 03:04:38

Add NFS service principal entry for the host in the AD server by running the setspn command.

For example, for nfs4ad.com domain, run the following command:

C:\Users\Administrator>setspn -A nfs/atsqa4-161.nfs4ad.com mapr
Checking domain DC=nfs4ad,DC=com
Registering ServicePrincipalNames for CN=mapr,CN=Users,DC=nfs4ad,DC=com
        nfs/atsqa4-164.nfs4ad.com
Updated object

Get the latest service ticket for the host from the AD server by running the kvno command.
For example:
```
# kvno nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM
nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM: kvno = 46
kvno nfs/qa108-43.nfs4ad.com@NFS4AD.COM
```

Add entry for NFS service principal key in the Kerberos keytab file, /etc/krb5.keytab:

# ktutil
ktutil:  addent -password -p nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM -k 46 -e RC4-HMAC
Ex: addent -password -p nfs/qa108-43.nfs4ad.com@NFS4AD.COM -k 46 -e RC4-HMAC
Password for nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM: 
( Give mapr user password i.e nfs4AD123 )
ktutil:  l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1   46      nfs/atsqa4-164.nfs4ad.com@NFS4AD.COM
ktutil:  wkt /etc/krb5.keytab 
ktutil:  q

Verify that NFS service principal and host principal are in the /etc/krb5.keytab file by running the klist command.

For example, for domain nfs4ad.com, run the following command and verify the entries in the file:

# klist -kt /etc/krb5.keytab 
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 host/atsqa4-161.nfs4ad.com@NFS4AD.COM
   4 08/01/2018 00:29:21 host/ATSQA4-161@NFS4AD.COM
   4 08/01/2018 00:29:21 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:21 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:22 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:22 ATSQA4-161$@NFS4AD.COM
   4 08/01/2018 00:29:22 ATSQA4-161$@NFS4AD.COM
  46 08/01/2018 02:58:01 nfs/atsqa4-161.nfs4ad.com@NFS4AD.COM

Ensure that /etc/krb5.keytab file is owned by user mapr and if necessary, change ownership to user mapr.

For example:

[root@qa108-41 ~]# chown mapr:root /etc/krb5.keytab 
[root@qa108-41 ~]# ls -l /etc/krb5.keytab 
-rw------- 1 mapr root 4175 Jul 22 23:53 /etc/krb5.keytab

Restart the rpcgssd service on the host to establish GSS security contexts.
CentOS
```
service rpcgssd start
```
Ubuntu
```
service gssd restart
```
Enable security variable, SecType, in the NFSv4 server configuration file at /opt/mapr/conf/nfs4server.conf.
For example:
```
  # Security type (krb5,krb5i,krb5p)
  SecType = krb5;
```
Start the NFSv4 server.
For more information, see Starting, Stopping, and Restarting HPE Ezmeral Data Fabric NFSv4.
List the shares exported on the server by running showmount -e command.
If the protocol is v4 only, the showmount command will not return the list of exported NFS shares. Instead, to view the export list, run the following command:
```
/opt/mapr/server/nfs4mgr list-exports
```
Ensure that the list-exports command runs successfully.
For example:
```
#  maprcli nfs4mgmt list-exports
Export Id     Path
30            /mapr
0             /
```

(Troubleshooting) Run the following command to restart the services if you see security-related issues.

CentOS

maprcli node services -nfs4 stop -nodes `hostname` ; service rpcgssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs restart ; service nfs stop ; sleep 2; maprcli node services -nfs4 start -nodes `hostname`

Ubuntu

maprcli node services -nfs4 stop -nodes `hostname` ; service gssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs-kernel-server restart ; service nfs-kernel-server stop ; sleep 2; maprcli node services -nfs4 start -nodes `hostname`

Set up VIPs for the NFSv4 servers:
1. Add entries for IPs and names of VIPs in the /etc/hosts file on the NFSv4 server host first and then on the AD server host.
  For example:
```
10.10.88.14 nfsvirtualip1
10.10.88.15 nfsvirtualip2
```
2. Add NFS service principal for the virtual IP by running the setspn command.
  For example:
```
C:\Users\Administrator>setspn -A host/nfsvirtualip1 nfsserver
C:\Users\Administrator>setspn -A nfs/nfsvirtualip1 nfsserver
                                    
C:\Users\Administrator>setspn -A host/nfsvirtualip2 nfsserver
C:\Users\Administrator>setspn -A nfs/nfsvirtualip2 nfsserver
```
3. Restart the rpcgssd service on the host to re-establish GSS security contexts.
  For example:
```
service rpcgssd restart
```

Configuring NFSv4 Server for Other Kerberos Installations

Configure NFS server for Kerberos.
Consult with your system administrator for assistance with the commands for configuring the NFS server for Kerberos-based authentication. For example, you must do the following:
- Create a service principal with nfs as the service name.
  For example: nfs/host.domain.com@REALM
- Generate a keytab for the NFS service principal, store it in the /etc/krb5.keytab file, and set correct permissions on the file.
Enable the security variable, SecType, in the NFSv4 server configuration file at /opt/mapr/conf/nfs4server.conf.
For example:
```
  # Security type (krb5,krb5i,krb5p)
  SecType = krb5;
```
Start the NFSv4 server.
For more information, see Starting, Stopping, and Restarting HPE Ezmeral Data Fabric NFSv4.
List the shares exported on the server by running showmount -e command.
If the protocol is v4 only, the showmount command will not return the list of exported NFS shares. Instead, to view the export list, run the following command:
```
/opt/mapr/server/nfs4mgr list-exports
```
Ensure that the list-exports command runs successfully.
For example:
```
#  maprcli nfs4mgmt list-exports
Export Id     Path
30            /mapr
0             /
```

Configuring NFSv4 Client

Ensure that NFS client has a /etc/krb5.keytab file with a valid principal similar to one of the following: nfs/<client_fqdn>@<domain>@<REALM>, host/<client_fqdn>@<domain>@<REALM>, or <HOSTNAME>$@<REALM>.
If the principal is not present, create the keytab file with the principal, which will be used to mount the share, for the OS (as mentioned in the OS vendor documentation).

Mount the cluster by running the mount command.

For example:

mount -t nfs4 -o sec=<security-type> <nfs4-server-hostname>:/<pseudo-path> <mount-point>

For example:

# mount -t nfs4 -o sec=krb5 <FQDN>:/mapr /mnt/nfs4mnt

Generate user ticket for the user to access the mount path.
For example, for user mapr on domain nfs4ad.com, run one of the following commands to generate the ticket:
- ```
kinit mapr@NFS4AD.COM
<Enter password>
```
- ```
echo usr2AD123 | kinit user2@NFS4AD.COM
```
Note: You must renew the user ticket before it expires; otherwise, the mount path returns permissions denied error after the ticket expires.
(Troubleshooting) Restart the services and mount again to avoid security-related issues.
CentOS
```
service rpcgssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs stop
```
Ubuntu
```
service rpcgssd restart; sleep 1; service rpcbind  restart ; sleep 1; service nfs stop
```
Troubleshooting: Any running IO on NFSv4 mount (with Kerberos) is stuck if the krb5 ticket expires for the current user. The mount point also hangs and becomes inaccessible.
Workaround: Restart the rpcgssd service with the new ticket to make the mount point accessible and re-trigger the IO to proceed.