Provides guidance on setting DVS configuration values but no details on how to use the configurator or edit configuration worksheets.
This procedure assumes that the user is either in a
configurator interactive session with the DVS service (
cray_dvs) selected or is editing the DVS configuration worksheet (
cray_dvs_worksheet.yaml).
The following steps correspond to the DVS configuration settings available through the configurator, and step numbering reflects the order in which those settings are presented when using the configurator in auto mode. They can be accessed in any order if using the configurator in interactive mode or if entering data in the configuration worksheet.
CAUTION: Configure DVS and all services only through the configurator or by placing/editing configuration files in the Simple Sync directory structure within the config set. Do not configure DVS by manually adding lines to /etc/fstab or other /etc files. In general, changes to those files are not persistent, and rebooting could result in loss of data.
Important: When configurator guidance indicates a relationship or interaction between one or more settings, it is advisory only; the configurator does not automatically check to ensure compatibility among settings. However, the underlying implementation of DVS is unchanged, and it does automatically set related mount options when certain mount options are specified. To prevent mount failure, enter setting values that are compatible, in accordance with the instructions in this publication.
- Enable Cray DVS service.
Ensure that cray_dvs.enabled is set to true.
- Configure a DVS client mount. Repeat this step to configure multiple client mounts, as needed.
A client mount defines the file system shares to be projected from DVS nodes to all (or selected) compute nodes. Each client mount is specified by a multival setting, which has a single key field followed by one or more other fields. The following substeps correspond to the fields (settings) of a client mount setting; substep numbering reflects the order in which these settings are presented when using the configurator in auto mode.
- Set the client mount reference.
reference | A human-readable string—a name—that is used to uniquely identify a client mount. reference cannot be set by accepting the default: a non-empty string is required. |
|---|
| Related settings/options: Because this is the key field of a client mount setting entry, each setting within the client mount setting includes this string in its full setting name. |
|---|
- Set the pathname of the mount point on the client.
mount_point | A string that specifies the full pathname on the client of the projected file system. mount_point cannot be set by accepting the default: a non-empty string is required. |
|---|
| Related settings/options: none |
|---|
- Set the pathname of the mount point on the DVS server node.
spath | A string that specifies the full pathname on the DVS server of the file system that is to be projected for a client mount. It must be an absolute path and it must exist on the DVS server. spath cannot be set by accepting the default: a non-empty string is required. |
|---|
| Related settings/options: none |
|---|
- Set the list of DVS server groups.
server_groups | A list of node groups that will function as DVS servers for a client mount. Enter one node group per line. server_groups cannot be set by accepting the default: a non-empty list is required.Important: DVS servers should be dedicated because they use unlimited amounts of CPU and memory resources based directly on the I/O requests sent from DVS clients. Avoid using nodes that have other services (Lustre nodes, login nodes, etc.) or are tier2 nodes. |
|---|
| Related settings/options: Functionally equivalent to the nodename or nodefile "additional" option in the options setting of the client mount setting. The use of those two additional options is deprecated. |
|---|
- Set the list of DVS client groups.
client_groups |
A list of node groups that will function as DVS clients for a client mount. Enter node groups one per line. Unlike server_groups, client_groups can be set to an empty list. If no node groups are specified, the mount will be performed on all suitable compute nodes (a compute node functioning as a DVS server is an example of an unsuitable node). This is common. |
|---|
| Related settings/options: none |
|---|
- Set the loadbalance option.
loadbalance | Used to specify loadbalance mode, which more evenly distributes loads across DVS servers. Loadbalance mode is valid only for read-only mounts. |
|---|
| Related settings/options: When loadbalance is enabled, the underlying DVS implementation automatically sets the readonly setting to true and sets these additional options: cache=1, failover=1, maxnodes=1, and hash_on_nid=1. Cray recommends setting the attrcache_timeout setting as well to take advantage of the mount being read-only. If loadbalance is enabled, leave the readonly setting unconfigured or set it to true to maintain consistency with the way DVS implements loadbalance.
|
|---|
- Set the attribute cache timeout option.
attrcache_timeout | Enables client-side attribute caching, which can significantly increase performance, most notably in pathname lookup situations.
When attribute caching is disabled, DVS clients must send a lookup request to a DVS server for every level of a pathname, and repeat this for every pathname operation. When it is enabled, it sends a lookup request to a DVS server for every level of a pathname once per n seconds. |
|---|
| The configurator default is 14400 seconds. The underlying DVS implementation default is 3 seconds, which is safer for read-write mounts. This means that to enhance system performance for read-only mounts, configure this setting by accepting the configurator default (or entering some other value). Leaving this setting unconfigured will result in the underlying default being used.Related settings/options: The Ansible play that consumes DVS configuration data prevents use of this mount option for read-write file systems due to the risk of file system corruption. Run-time mounts not accompanied by that Ansible play do not have that safeguard. In such cases, if a read-write mount is created, it is safe to leave attrcache_timeout unconfigured so that the underlying default is used. |
|---|
- Set the read-only option.
readonly | Determines whether the client mount is read-only or read-write. If intending to enable client-side caching of read data on a non-writable file system, use this readonly setting to force the DVS mount to be read-only. This will disable write caching. |
|---|
| The configurator default is true or 1. The underlying DVS implementation default is false. Leaving this setting unconfigured will result in the underlying DVS default being used.Related settings/options: When loadbalance is enabled, DVS automatically enables readonly but the configurator does not, so either leave this setting unconfigured or accept the configurator default. If the attrcache_timeout setting is set for this client mount, readonly should be enabled (set to true) in the configurator/worksheet. If the cache option is specified in the options setting for this client mount, enabling readonly is the only way to enable read caching without enabling write caching as well). |
|---|
- Set other client mount options.
options | Provides the only way to specify mount options in addition to those already specified in the other mount point settings. Enter a string with mount options separated by comma and no spaces. For information about available options and their implications, see "Additional Options for Use in the Options Setting of a Client Mount." Note that it is necessary to specify maxnodes=1 here for a read-write client mount of an NFS or other non-cluster, non-coherent file system. |
|---|
| Related settings/options: Options contained in this setting will be appended to the mount options specified in other settings. Any that are functionally redundant with settings already configured (such as nodename/nodefile, which are redundant with the server_groups setting) will override those settings. |
|---|
This completes the entry of a client mount setting. Review the setting fields in the configurator summary or configuration worksheet. Revisit any of the settings to make changes, if needed. Repeat
Step 2: Configure a DVS client mount to specify additional client mounts.
- Set DVS kernel module parameters.
Two DVS kernel module parameters can be set using the configurator by following the substeps below. Note that changing these parameters after initial configuration may require reloading the module to enable the change to take effect. To change other kernel module parameters, see
Configure DVS using Modprobe or Proc Files.
- Set the DVS interprocess communication (IPC) heartbeat timeout.
dvsipc_heartbeat_timeout | DVS inter-process communication (IPC) heartbeat timeout, in seconds. This parameter is no longer used; it has been preserved only to maintain backwards compatibility with existing DVS config files. Leave this parameter unconfigured or accept the default value. |
|---|
| Related settings/options: none |
|---|
- When done configuring DVS, take one of the following actions:
| If configuring DVS | Do this |
|---|
| interactively | Save the changes and exit the configurator session.Cray DVS Service Menu [default: save & exit - Q] $ Q cfgset will save all of the configuration settings in the config set (in configuration worksheets and templates), run post-configuration scripts, create a time-stamped clone of the config set, and exit automatically. |
|---|
| using the worksheet | Import the completed DVS worksheet by updating the config set and specifying the worksheet path.smw# cfgset update --worksheet-path \
'/some/edit/location/cray_dvs_worksheet.yaml' p0
|
|---|
- Proceed to Validate the Config Set and Run Ansible Plays.