Configure DVS using Worksheets

How to configure DVS using configuration worksheets.

This procedure assumes that the user has generated configuration worksheets and is editing the DVS configuration worksheet (cray_dvs_worksheet.yaml). If new worksheets need to be generated, use this procedure:

Generate up-to-date worksheets for config set p0 (merges any new service packages installed on the system with data already in config set p0).
```
smw# cfgset update --mode prepare --no-scripts p0
```
Make a copy of the CLE configuration worksheets directory outside the config set to be used as a work area for editing. The worksheets should not be edited in their original location for two reasons: (1) the configurator will not permit updating a config set from worksheets within that config set, and (2) edits would be overwritten when the config set is updated.
```
smw# cp /var/opt/cray/imps/config/sets/p0/worksheets/* \
/var/adm/cray/release/p0_worksheet_workarea
```
(This is the same work area used in an initial installation of SMW/CLE software.

Edit the DVS worksheet:

smw# vi /var/adm/cray/release/p0_worksheet_workarea/cray_dvs_worksheet.yaml

The following steps correspond to the configuration settings available in the DVS worksheet, and step numbering reflects the order in which those settings appear there.

CAUTION: Configure DVS and all services only through the configurator or by placing/editing configuration files in the Simple Sync directory structure within the config set. Do not configure DVS by manually adding lines to /etc/fstab or other /etc files. In general, changes to those files are not persistent, and rebooting could result in loss of data.

Important: When configurator guidance indicates a relationship or interaction between one or more settings, it is advisory only; the configurator does not automatically check to ensure compatibility among settings. However, the underlying implementation of DVS is unchanged, and it does automatically set related mount options when certain mount options are specified. To prevent mount failure, enter setting values that are compatible, in accordance with the instructions in this publication.

Enable Cray DVS service.

Uncomment the line that sets the cray_dvs.enabled setting, and ensure that it is set to true.

# Enable 'cray_dvs' Service? (boolean, level=basic)
cray_dvs.enabled: true
#
#**************** END Service Enable/Disable ****************

Configure a client mount. Repeat this step to configure multiple mounts, as needed.

This step defines the file system shares to be projected from DVS nodes to all or selected compute nodes. Each client mount entry is a multival setting, which means it has a single key field followed by one or more other fields.

In the worksheet, copy the nine lines below # ** EXAMPLE 'client_mount' VALUE (with current defaults) ** and paste them below the line # NOTE: Place additional 'client_mount' setting entries here, if desired. Repeat this for each client mount.

# ** EXAMPLE 'client_mount' VALUE (with current defaults) **
#   cray_dvs.settings.client_mount.data.reference.sample_key_a: null   <-- setting a multival key
#   cray_dvs.settings.client_mount.data.sample_key_a.mount_point: ''
#   cray_dvs.settings.client_mount.data.sample_key_a.spath: ''
#   cray_dvs.settings.client_mount.data.sample_key_a.server_groups: []
#   cray_dvs.settings.client_mount.data.sample_key_a.client_groups: []
#   cray_dvs.settings.client_mount.data.sample_key_a.loadbalance: false
#   cray_dvs.settings.client_mount.data.sample_key_a.attrcache_timeout: 14400
#   cray_dvs.settings.client_mount.data.sample_key_a.readonly: true
#   cray_dvs.settings.client_mount.data.sample_key_a.options: ''

Uncomment the lines and remove the <-- setting a multival key text at the end of the first line (note that the null value is required; do not remove or change it). Finally, modify the values as needed for this site. The substeps that follow provide guidance for changing or keeping the default value for each field. To leave a setting unconfigured, keep/restore the comment symbol at the beginning of the line.

Set the client mount reference.

`reference`	A human-readable string—a name—that is used to uniquely identify a client mount. `reference` cannot be set by accepting the default: a non-empty string is required. Replace `sample_key_a` with the chosen string (`REF-NAME` in example below) in all lines. Do not change the `null` value in the first line.
	Related settings/options: Because this is the key field of a client mount setting entry, each setting within the client mount setting includes this string in its full setting name.

# NOTE: Place additional 'client_mount' setting entries here, if desired.
cray_dvs.settings.client_mount.data.reference.REF-NAME: null
cray_dvs.settings.client_mount.data.REF-NAME.mount_point: ''
cray_dvs.settings.client_mount.data.REF-NAME.spath: ''
cray_dvs.settings.client_mount.data.REF-NAME.server_groups: []
cray_dvs.settings.client_mount.data.REF-NAME.client_groups: []
cray_dvs.settings.client_mount.data.REF-NAME.loadbalance: false
cray_dvs.settings.client_mount.data.REF-NAME.attrcache_timeout: 14400
cray_dvs.settings.client_mount.data.REF-NAME.readonly: true
cray_dvs.settings.client_mount.data.REF-NAME.options: ''
#**************** END Service Setting: client_mount ****************

Set the pathname of the mount point on the client.

`mount_point`	A string that specifies the full pathname on the client of the projected file system. `mount_point` cannot be set by accepting the default: a non-empty string is required. Replace the default empty string with a full pathname.
	Related settings/options: none

cray_dvs.settings.client_mount.data.REF-NAME.mount_point: /CLIENT-PATH

Set the pathname of the mount point on the DVS server node.

`spath`	A string that specifies the full pathname on the DVS server of the file system that is to be projected for a client mount. It must be an absolute path and it must exist on the DVS server. `spath` cannot be set by accepting the default: a non-empty string is required. Replace the default empty string with a pathname that is an absolute path that exists on the DVS server.
	Related settings/options: none

cray_dvs.settings.client_mount.data.REF-NAME.spath: /DVS-PATHNAME

Set the list of DVS server nodes.

`server_groups`	A list of node groups that will function as DVS servers for a client mount. Enter one node group per line. `server_groups` cannot be set by accepting the default: a non-empty list is required. Important: DVS servers should be dedicated because they use unlimited amounts of CPU and memory resources based directly on the I/O requests sent from DVS clients. Avoid using nodes that have other services (Lustre nodes, login nodes, etc.) or are tier2 nodes. Replace the default empty list with one or more node groups; if no node groups are entered, no mount is made.
	Related settings/options: Functionally equivalent to the `nodename` or `nodefile` "additional" option in the `options` setting of the client mount setting. The use of those two additional options is deprecated.

server_groups

A list of node groups that will function as DVS servers for a client mount. Enter one node group per line. server_groups cannot be set by accepting the default: a non-empty list is required.

Important: DVS servers should be dedicated because they use unlimited amounts of CPU and memory resources based directly on the I/O requests sent from DVS clients. Avoid using nodes that have other services (Lustre nodes, login nodes, etc.) or are tier2 nodes.

Replace the default empty list with one or more node groups; if no node groups are entered, no mount is made.

Related settings/options: Functionally equivalent to the nodename or nodefile "additional" option in the options setting of the client mount setting. The use of those two additional options is deprecated.

cray_dvs.settings.client_mount.data.REF-NAME.server_groups:
node-group-1
node-group-2

Set the list of DVS client nodes.

`client_groups`	A list of node groups that will function as DVS clients for a client mount. Enter node groups one per line. Unlike `server_groups`, `client_groups` can be set to an empty list. If no node groups are specified, the mount will be performed on all suitable compute nodes (a compute node functioning as a DVS server is an example of an unsuitable node). This is common.
	Related settings/options: none

cray_dvs.settings.client_mount.data.REF-NAME.client_groups:
node-group-3

Set the loadbalance option.

loadbalance Used to specify loadbalance mode, which more evenly distributes loads across DVS servers. Loadbalance mode is valid only for read-only mounts.

Related settings/options: When loadbalance is enabled, the underlying DVS implementation automatically sets the readonly setting to true and sets these additional options: cache=1, failover=1, maxnodes=1, and hash_on_nid=1. Cray recommends setting the attrcache_timeout setting as well to take advantage of the mount being read-only. If loadbalance is enabled, leave the readonly setting unconfigured or set it to true to maintain consistency with the way DVS implements loadbalance.

`loadbalance`	Used to specify loadbalance mode, which more evenly distributes loads across DVS servers. Loadbalance mode is valid only for read-only mounts.
	Related settings/options: When `loadbalance` is enabled, the underlying DVS implementation automatically sets the `readonly` setting to `true` and sets these additional options: `cache=1`, `failover=1`, `maxnodes=1`, and `hash_on_nid=1`. Cray recommends setting the `attrcache_timeout` setting as well to take advantage of the mount being read-only. If `loadbalance` is enabled, leave the `readonly` setting unconfigured or set it to `true` to maintain consistency with the way DVS implements `loadbalance`.

cray_dvs.settings.client_mount.data.REF-NAME.loadbalance: true

Set the attribute cache timeout option.

attrcache_timeout Enables client-side attribute caching, which can significantly increase performance, most notably in pathname lookup situations. When attribute caching is disabled, DVS clients must send a lookup request to a DVS server for every level of a pathname, and repeat this for every pathname operation. When it is enabled, it sends a lookup request to a DVS server for every level of a pathname once per n seconds.

`attrcache_timeout`	Enables client-side attribute caching, which can significantly increase performance, most notably in pathname lookup situations. When attribute caching is disabled, DVS clients must send a lookup request to a DVS server for every level of a pathname, and repeat this for every pathname operation. When it is enabled, it sends a lookup request to a DVS server for every level of a pathname once per `n` seconds.
	The configurator default is `14400` seconds. The underlying DVS implementation default is `3` seconds, which is safer for read-write mounts. This means that to enhance system performance for read-only mounts, configure this setting by accepting the configurator default (or entering some other value). Leaving this setting unconfigured will result in the underlying default being used. Related settings/options: The Ansible play that consumes DVS configuration data prevents use of this mount option for read-write file systems due to the risk of file system corruption. Run-time mounts not accompanied by that Ansible play do not have that safeguard. In such cases, if a read-write mount is created, it is safe to leave `attrcache_timeout` unconfigured so that the underlying default is used.

The configurator default is 14400 seconds. The underlying DVS implementation default is 3 seconds, which is safer for read-write mounts. This means that to enhance system performance for read-only mounts, configure this setting by accepting the configurator default (or entering some other value). Leaving this setting unconfigured will result in the underlying default being used.

Related settings/options: The Ansible play that consumes DVS configuration data prevents use of this mount option for read-write file systems due to the risk of file system corruption. Run-time mounts not accompanied by that Ansible play do not have that safeguard. In such cases, if a read-write mount is created, it is safe to leave attrcache_timeout unconfigured so that the underlying default is used.

cray_dvs.settings.client_mount.data.REF-NAME.attrcache_timeout: 14400

Set the read-only option.

readonly Determines whether the client mount is read-only or read-write. If intending to enable client-side caching of read data on a non-writable file system, use this readonly setting to force the DVS mount to be read-only. This will disable write caching.

`readonly`	Determines whether the client mount is read-only or read-write. If intending to enable client-side caching of read data on a non-writable file system, use this `readonly` setting to force the DVS mount to be read-only. This will disable write caching.
	The configurator default is `true` or `1`. The underlying DVS implementation default is `false`. Leaving this setting unconfigured will result in the underlying DVS default being used. Related settings/options: When `loadbalance` is enabled, DVS automatically enables `readonly` but the configurator does not, so either leave this setting unconfigured or accept the configurator default. If the `attrcache_timeout` setting is set for this client mount, `readonly` should be enabled (set to true) in the configurator/worksheet. If the `cache` option is specified in the `options` setting for this client mount, enabling `readonly` is the only way to enable read caching without enabling write caching as well).

The configurator default is true or 1. The underlying DVS implementation default is false. Leaving this setting unconfigured will result in the underlying DVS default being used.

Related settings/options: When loadbalance is enabled, DVS automatically enables readonly but the configurator does not, so either leave this setting unconfigured or accept the configurator default. If the attrcache_timeout setting is set for this client mount, readonly should be enabled (set to true) in the configurator/worksheet. If the cache option is specified in the options setting for this client mount, enabling readonly is the only way to enable read caching without enabling write caching as well).

cray_dvs.settings.client_mount.data.REF-NAME.readonly: true

Set other client mount options.

options Provides the only way to specify mount options in addition to those already specified in the other mount point settings. Enter a string with mount options separated by comma and no spaces. For information about available options and their implications, see "Additional Options for Use in the Options Setting of a Client Mount." Note that it is necessary to specify maxnodes=1 here for a read-write client mount of an NFS or other non-cluster, non-coherent file system.

Related settings/options: Options contained in this setting will be appended to the mount options specified in other settings. Any that are functionally redundant with settings already configured (such as nodename/nodefile, which are redundant with the server_groups setting) will override those settings.

`options`	Provides the only way to specify mount options in addition to those already specified in the other mount point settings. Enter a string with mount options separated by comma and no spaces. For information about available options and their implications, see "Additional Options for Use in the Options Setting of a Client Mount." Note that it is necessary to specify `maxnodes=1` here for a read-write client mount of an NFS or other non-cluster, non-coherent file system.
	Related settings/options: Options contained in this setting will be appended to the mount options specified in other settings. Any that are functionally redundant with settings already configured (such as `nodename`/`nodefile`, which are redundant with the `server_groups` setting) will override those settings.

In the example, this client mount point is being set to atomic stripe parallel mode.

cray_dvs.settings.client_mount.data.REF-NAME.options: maxnodes=3,atomic

This completes the entry of a client mount setting. Review the setting fields and make changes, if needed. Repeat step 2 to configure additional client mount points.

Set DVS kernel module parameters.

Two DVS kernel module parameters can be set in this worksheet by following the substeps below. Note that changing these parameters after initial configuration may require reloading the module to enable the change to take effect. To change other kernel module parameters, see Configure DVS using Modprobe or Proc Files.

Set the DVS interprocess communication (IPC) heartbeat timeout.

`dvsipc_heartbeat_timeout`	DVS inter-process communication (IPC) heartbeat timeout, in seconds. This parameter is no longer used; it has been preserved only to maintain backwards compatibility with existing DVS config files. Leave this parameter unconfigured or accept the default value.
	Related settings/options: none

# ---------------- kernel_param : dvsipc_heartbeat_timeout ----------------
#
#cray_dvs.settings.kernel_param.data.dvsipc_heartbeat_timeout: 60

Set the DVS debug mask.

# ---------------- kernel_param : dvs_debug_mask ----------------
#
#cray_dvs.settings.kernel_param.data.dvs_debug_mask: 0

Import the completed DVS worksheet by updating the config set and specifying the worksheet path.
```
smw# cfgset update --worksheet-path \
'/some/edit/location/cray_dvs_worksheet.yaml' p0
```
When the config set is updated using the configurator, all of the pre-and post-configuration scripts are run.
Remember: When importing worksheets using cfgset with the --worksheet-path option,
- Always add single quote marks around the worksheet path if a wildcard is used (e.g.,*_worksheet.yaml).
- Do not add mode, state, level, or service options; the configurator ignores them for worksheet import.
- The type of the config set must match the type of the worksheets being imported.
Proceed to Validate the Config Set and Run Ansible Plays.