Reconfigure DVS Interactively

How to reconfigure DVS using the configurator in interactive mode.

This procedure assumes that a config set has been created and DVS is already configured.

This procedure describes how to reconfigure DVS by updating the DVS service in config set p0. To use the commands in the examples, replace p0 with the name of the config set being updated.
CAUTION: Configure DVS and all services only through the configurator or by placing/editing configuration files in the Simple Sync directory structure within the config set. Do not configure DVS by manually adding lines to /etc/fstab or other /etc files. In general, changes to those files are not persistent, and rebooting could result in loss of data.
Important: When configurator guidance indicates a relationship or interaction between one or more settings, it is advisory only; the configurator does not automatically check to ensure compatibility among settings. However, the underlying implementation of DVS is unchanged, and it does automatically set related mount options when certain mount options are specified. To prevent mount failure, enter setting values that are compatible, in accordance with the instructions in this publication.
  1. (Optional) Use the cfgset command to view current DVS settings.
    smw# cfgset search --service cray_dvs --level advanced p0
  2. Use cfgset to update DVS.
    To reconfigure DVS interactively:
    smw# cfgset update -s cray_dvs -l advanced --mode interactive p0
    The cfgset command invokes the configurator in interactive mode, and the configurator presents the Cray DVS Service Menu, which displays all DVS settings and their values.
    Service Configuration Menu (Config Set : p0, type: cle)

cray_dvs          { status: enabled }  { validation: valid }
--------------------------------------------------------------
 Selected   #    Settings           Value/Status (level=advanced)
--------------------------------------------------------------
            1)   client_mount
                    reference: CSS            [OK]
                    reference: ComputeHome    [OK]

                 kernel_param
            2)      dvsipc_heartbeat_timeout  60
            3)      dvs_debug_mask            0

...
Cray DVS Service Menu [default: save & exit - Q] $
  3. Make changes to an existing client mount (client_mount) or kernel module parameter (kernel_param), or add a new client mount.
    • To make changes to an existing client mount, enter 1 at the prompt and press <cr> (Enter), then enter C and press <cr> to display the configuration setting screen. 
      Service Configuration Menu (Config Set : p0, type: cle)
...
Cray DVS Service Menu [default: save & exit - Q] $ 1
...
Cray DVS Service Menu [default: configure - C] $ C
*********** cray_dvs.settings.client_mount **************

     client_mount
       <guidance>

     Configured Values:
         1) 'CSS'
             a) mount_point: /cray
             b) spath: /cray
             c) server_groups: 
                    dvs_nodes
             d) client_groups: (none)
             e) loadbalance: False
             f) attrcache_timeout: 3
             g) readonly: False
             h) options: maxnodes=1

         2) 'ComputeHome'
             a) mount_point: /home
             b) spath: /home
             c) server_groups: 
                    dvs_nodes
             d) client_groups: (none)
             e) loadbalance: False
             f) attrcache_timeout: 3 
             g) readonly: False
             h) options: maxnodes=1

cray_dvs.settings.client_mount
[<cr>=set 2 entries, +=add an entry, ?=help, @=less] $
      In the configuration setting screen:
      • To modify the setting values of an existing client mount, enter the number of the client mount to be modified and the letter of the setting to be modified, followed by an asterisk. In this example, the second client mount is selected and the setting corresponding to list item 'b' is selected (the spath setting).
        cray_dvs.settings.client_mount
[<cr>=set 2 entries, +=add an entry, ?=help, @=less] $ 2b*
      • To delete a client mount, enter the number of the client mount to be deleted, followed by a minus sign. In this example, the second client mount is deleted.
        cray_dvs.settings.client_mount
[<cr>=set 2 entries, +=add an entry, ?=help, @=less] $ 2-
    • To set or change one of the kernel module parameters listed in the Cray DVS Service Menu, enter the number of the parameter to be set/changed at the prompt and press <cr>, then enter C and press <cr> to display the configuration setting screen. In this example, setting 2 is selected, which is the first kernel parameter. To change kernel module parameters that do not appear in the Cray DVS Service Menu, see Configure DVS using Modprobe or Proc Files. 
      Cray DVS Service Menu [default: save & exit - Q] $ 2
...
Cray DVS Service Menu [default: configure - C] $ C
...
cray_dvs.settings.kernel_param.data.dvsipc_heartbeat_timeout
[<cr>=keep'60', <new value>, ?=help, @=less] $
      In the configuration setting screen, accept the current value (<cr>), revert to the default value (#), or enter a new value. Enter ? to see a list of possible commands.
    • To add a new client mount, follow these instructions:
    1. Select the client mount setting.
      Cray DVS Service Menu [default: save & exit - Q] $ 1
...
Cray DVS Service Menu [default: configure - C] $ C
    2. Add a new client mount entry.
      cray_dvs.settings.client_mount
[<cr>=set 2 entries, +=add an entry, ?=help, @=less] $ +
    3. Set the client mount reference.
      referenceA human-readable string—a name—that is used to uniquely identify a client mount. reference cannot be set by accepting the default: a non-empty string is required.
      Related settings/options: Because this is the key field of a client mount setting entry, each setting within the client mount setting includes this string in its full setting name.
      In this example, the string tmp_failover is used as the client mount reference. Whatever is chosen as the reference is used in the full setting names of all of the rest of that mount's settings.
      cray_dvs.settings.client_mount.data.reference
[<cr>=set '', <new value>, ?=help, @=less] $ tmp_failover
    4. Set the pathname of the mount point on the client.
      mount_pointA string that specifies the full pathname on the client of the projected file system. mount_point cannot be set by accepting the default: a non-empty string is required.
      Related settings/options: none
      cray_dvs.settings.client_mount.data.tmp_failover.mount_point
[<cr>=set '', <new value>, ?=help, @=less] $ /here
    5. Set the pathname of the mount point on the DVS server node.
      spathA string that specifies the full pathname on the DVS server of the file system that is to be projected for a client mount. It must be an absolute path and it must exist on the DVS server. spath cannot be set by accepting the default: a non-empty string is required.
      Related settings/options: none
      cray_dvs.settings.client_mount.data.tmp_failover.spath
[<cr>=set '', <new value>, ?=help, @=less] $ /tmp
    6. Set the list of DVS server nodes.
      server_groupsA list of node groups that will function as DVS servers for a client mount. Enter one node group per line. server_groups cannot be set by accepting the default: a non-empty list is required.
      Important: DVS servers should be dedicated because they use unlimited amounts of CPU and memory resources based directly on the I/O requests sent from DVS clients. Avoid using nodes that have other services (Lustre nodes, login nodes, etc.) or are tier2 nodes.
      Related settings/options: Functionally equivalent to the nodename or nodefile "additional" option in the options setting of the client mount setting. The use of those two additional options is deprecated.
      cray_dvs.settings.client_mount.data.tmp_failover.servers
[<cr>=set 0 entries, +=add an entry, ?=help, @=less] $ +
Add servers (Ctrl-d to exit) $ c0-0c2s1n1
Add servers (Ctrl-d to exit) $ c0-0c1s2n1
Add servers (Ctrl-d to exit) $ <Ctrl-d>
     2 entries added. Press <cr> to set.
cray_dvs.settings.client_mount.data.tmp_failover.servers
[<cr>=set 2 entries, +=add an entry, ?=help, @=less] $ <cr>
    7. Set the list of DVS clients.
      client_groups A list of node groups that will function as DVS clients for a client mount. Enter node groups one per line. Unlike server_groups, client_groups can be set to an empty list. If no node groups are specified, the mount will be performed on all suitable compute nodes (a compute node functioning as a DVS server is an example of an unsuitable node). This is common.
      Related settings/options: none
      cray_dvs.settings.client_mount.data.tmp_failover.clients
[<cr>=set 0 entries, +=add an entry, ?=help, @=less] $ <cr>
      This example sets clients to an empty list.
    8. Set the loadbalance option.
      loadbalanceUsed to specify loadbalance mode, which more evenly distributes loads across DVS servers. Loadbalance mode is valid only for read-only mounts.
      Related settings/options: When loadbalance is enabled, the underlying DVS implementation automatically sets the readonly setting to true and sets these additional options: cache=1, failover=1, maxnodes=1, and hash_on_nid=1. Cray recommends setting the attrcache_timeout setting as well to take advantage of the mount being read-only. If loadbalance is enabled, leave the readonly setting unconfigured or set it to true to maintain consistency with the way DVS implements loadbalance. 
      cray_dvs.settings.client_mount.data.tmp_failover.loadbalance
[<cr>=set 'false', <new value>, ?=help, @=less] $ <cr>
      Pressing <cr> accepts the configurator default, which is to not enable loadbalance.
    9. Set the attribute cache timeout option.
      attrcache_timeoutEnables client-side attribute caching, which can significantly increase performance, most notably in pathname lookup situations. When attribute caching is disabled, DVS clients must send a lookup request to a DVS server for every level of a pathname, and repeat this for every pathname operation. When it is enabled, it sends a lookup request to a DVS server for every level of a pathname once per n seconds.
      The configurator default is 14400 seconds. The underlying DVS implementation default is 3 seconds, which is safer for read-write mounts. This means that to enhance system performance for read-only mounts, configure this setting by accepting the configurator default (or entering some other value). Leaving this setting unconfigured will result in the underlying default being used.

      Related settings/options: The Ansible play that consumes DVS configuration data prevents use of this mount option for read-write file systems due to the risk of file system corruption. Run-time mounts not accompanied by that Ansible play do not have that safeguard. In such cases, if a read-write mount is created, it is safe to leave attrcache_timeout unconfigured so that the underlying default is used.

      cray_dvs.settings.client_mount.data.tmp_failover.attrcache_timeout
[<cr>=set '14400', <new value>, ?=help, @=less] $ <cr>
      Pressing <cr> accepts the configurator default of 14400, which will be fine because the next setting will make this a read-only mount.
    10. Set the read-only option.
      readonlyDetermines whether the client mount is read-only or read-write. If intending to enable client-side caching of read data on a non-writable file system, use this readonly setting to force the DVS mount to be read-only. This will disable write caching.
      The configurator default is true or 1. The underlying DVS implementation default is false. Leaving this setting unconfigured will result in the underlying DVS default being used.

      Related settings/options: When loadbalance is enabled, DVS automatically enables readonly but the configurator does not, so either leave this setting unconfigured or accept the configurator default. If the attrcache_timeout setting is set for this client mount, readonly should be enabled (set to true) in the configurator/worksheet. If the cache option is specified in the options setting for this client mount, enabling readonly is the only way to enable read caching without enabling write caching as well).

      cray_dvs.settings.client_mount.data.tmp_failover.readonly
[<cr>=set 'true', <new value>, ?=help, @=less] $ <cr>
    11. Set other client mount options.
      optionsProvides the only way to specify mount options in addition to those already specified in the other mount point settings. Enter a string with mount options separated by comma and no spaces. For information about available options and their implications, see "Additional Options for Use in the Options Setting of a Client Mount." Note that it is necessary to specify maxnodes=1 here for a read-write client mount of an NFS or other non-cluster, non-coherent file system.
      Related settings/options: Options contained in this setting will be appended to the mount options specified in other settings. Any that are functionally redundant with settings already configured (such as nodename/nodefile, which are redundant with the server_groups setting) will override those settings.
      cray_dvs.settings.client_mount.data.tmp_failover.options
[<cr>=set '', <new value>, ?=help, @=less] $ failover,cache
...
cray_dvs.settings.client_mount
[<cr>=set 3 entries, +=add an entry, ?=help, @=less] $ <cr>
      Suppose a desired option was omitted. Because the options setting is a string rather than a list, another option can be added only by entering a new string with all desired options, which replaces the old string. Revisit that setting (enter 1 at the service menu prompt and press <cr> twice, then enter 3h*, where 3 is the number of the mount entry, at the configuration setting screen prompt) and enter the new string of options.
      cray_dvs.settings.client_mount.data.tmp_failover.options
[<cr>=keep 'failover,cache', <new value>, ?=help, @=less] $ failover,cache,maxnodes=3
  4. Save the changes and exit the configurator session when done making changes.
    Cray DVS Service Menu [default: save & exit - Q] $ Q
    cfgset will save all of the configuration settings in the config set (in configuration worksheets and templates), run post-configuration scripts, create a time-stamped clone of the config set, and exit automatically.
  5. Proceed to Validate the Config Set and Run Ansible Plays.