Prepare the Local Configuration

Notes to prepare local Lustre configuration for ClusterStor 3.2

The ClusterStor system uses puppet-generated files for the Lustre/LNet config:

/etc/modprobe.d/lnet.conf

/etc/lustre/ip2nets.dat (OPTIONAL)

/etc/lustre/routes.dat (OPTIONAL)

The ip2nets and routes details can be included in the lnet config file or placed in separate files.

The lnet.conf file is used to configure LNet and related Lustre parameters. The routes and ip2nets configuration entries can be included in the lnet.conf file or placed in separate files. Larger configurations—where the character count for the ip2nets or routes entry exceeds the modprobe limit of 1024—must use separate files for ip2nets and/or routes.

Create a subdirectory in ~admin on management node 0 to store the custom configuration files, for example, ~admin/lnet_conf. Create a custom lnet.config file—as well as ip2nets and routes files, if necessary—then copy them into ~admin/lnet_conf.

Example lnet.config file:

## ko2iblnd parameters
options ko2iblnd timeout=10
options ko2iblnd peer_timeout=0
options ko2iblnd keepalive=30
options ko2blnd credits=2048
options ko2iblnd ntx=2048
### NOTE peer_credits must be consistent across all peers on the IB network
options ko2iblnd peer_credits=16
options ko2iblnd concurrent_sends=16

## LNet parameters
options lnet ip2nets="o2ib3(ib0) 10.10.100.*"
options lnet routes="gni 10.10.100.[60,61,62,63,64,65,66,67]@o2ib3"
options lnet router_ping_timeout=10
options lnet alive_router_check_interval=35

## Sonexion only (if off by default)
options lnet avoid_asym_router_failure=1

## ptlrpc parameters
options ptlrpc at_max=400
options ptlrpc at_min=40
options ptlrpc ldlm_enqueue_min=260

The ip2nets entry assigns the designated o2ib LNet index to the servers with the corresponding IP addrs.

The routes entry sets up routes to the corresponding LNet network, 'gni' in this case. The hosts at 10.10.100.[100,101,102] are LNet routers connected to both o2ib3 and the gni networks.

In some configurations, there may be hosts on the InfiniBand fabric with more than one connection to the fabric.

Those connections should be configured to use separate IP subnets. For LNet to properly handle such a configuration, IP-Aliasing is used on interface ib0 to configure additional IP subnets on the Lustre server nodes. This can be managed in the ip2nets specification by appropriately assigning lnet nids to the ib aliases, and in the modprobe lnet.config file by setting up IB aliases at module load time. Note that for systems using fine-grained routing, there may be separate subnets used for MDS vs. OSS nodes.

Examples of ip2nets and modprobe config entries for adding IP addresses to the ib0 alias on the servers (10.10.102 on mds nodes, 10.10.101 on OSS nodes):

Note: For ip2nets:

o2ib1 10.10.100.*
o2ib1000(ib0:1) 10.10.102.*
o2ib1002(ib0) 10.10.100.*
o2ib1003(ib0:1) 10.10.101.*
o2ib1004(ib0) 10.10.8.*
o2ib1005(ib0:1) 10.10.101.*

For lnet.config, these two lines can be added at the start of the local lnet.config file. The first line creates the ib0:1 aliases on lnet module load, the second line removes the aliases when lnet is unloaded:

Note:

install lnet if nodeattr mds; then /sbin/ip -o -4 a show ib0 | /usr/bin/awk '/inet/{s=$4;sub("10\\.10\\.100\\.","10.10.102.",s);print "/sbin/ip address add dev ib0 label ib0:1",s}' | /bin/sh ; else /sbin/ip -o -4 a show ib0 | /usr/bin/awk '/inet/{s=$4;sub("10\\.10\\.100\\.","10.10.101.",s);print "/sbin/ip address add dev ib0 label ib0:1",s}' | /bin/sh ; fi ; /sbin/ modprobe --ignore-install lnet $CMDLINE_OPTS

remove lnet /sbin/modprobe -r --ignore-remove lnet && /sbin/ip -o -4 a show label ib0:1 | awk '{print "/sbin/ip address del dev ib0 label ib0:1",$4}' | /bin/sh

For recommended parameter settings for the Lustre server nodes, see Lustre Server: Recommended Tuning Parameters. For instructions to generate the config files, see Create Custom LNet Configuration for ClusterStor.