Configure DVS using Modprobe or Proc Files

How to change DVS kernel module parameters either prior to boot or dynamically.

Most DVS kernel module parameters cannot be changed using the configurator. These parameters are typically changed by adding lines to a modprobe.d configuration file or echoing values to a /proc file. Changes to a modprobe.d file are made prior to booting the affected nodes, and the changes take effect at boot. To ensure that these changes persist across boots, the modprobe.d files can be placed in the Simple Sync directory of the working CLE config set. Changes made to those files will then be propagated to all target nodes using the Simple Sync mechanism.

The dvs.conf file is one of many files that are generated automatically and controlled by the Cray Configuration Management Framework (CMF). Such files can be identified by the warning statement in the file header, which includes a statement that the file should not be modified directly or updated using Simple Sync. For this reason, Cray recommends that sites create a local DVS configuration file (dvs-local.conf) for configuring the kernel module parameters listed in this section. Then dvs-local.conf can be placed in the appropriate Simple Sync directory to augment the configuration specified in dvs.conf.

The Simple Sync Mechanism and File Structure

Cray Simple Sync provides a generic mechanism to automatically distribute files to targeted locations on a Cray XC system. This mechanism can be used to override or change default system behavior through the contents of the distributed files. When enabled, the Simple Sync service is executed on all internal CLE nodes and eLogin nodes at boot time and whenever the administrator executes /etc/init.d/cray-ansible start on a CLE node or eLogin node. When Simple Sync is executed, files placed in the following directory structure are copied to the root file system (/) on the target nodes.

The Simple Sync directory structure has this root: smw:/var/opt/cray/imps/config/sets/<config_set>/files/simple_sync/

Below that root are the directories listed on the left. Files placed in those directories are copied to their associated target nodes.

Files placed here	are copied to
./common/files/	all internal and eLogin nodes
./platform/[compute\|service]/files/	all CLE compute nodes or all service nodes (not applicable to eLogin nodes)
./hardwareid/`<hardwareid>`/files/	nodes with matching hardware ID, which is the cname of a CLE node or the output of the hostid command (e.g., 1eac0b0c) on other nodes (not applicable to eLogin nodes)
./hostname/`<hostname>`/files/	nodes with matching host name (use this for eLogin nodes ONLY)
./nodegroups/`<node_group_name>`/files/	nodes in the matching node group

Note: The directory structure for a particular hardware ID or host name (everything below./hardwareid/ and ./hostname/) must be created manually as needed. This is unnecessary for node groups because their associated directories are created automatically by post-configuration callback scripts when the config set is created or updated using cfgset.

Anything (directory structure and files) placed below ./files/ in the Simple Sync directory structure on the SMW is replicated on the target node starting at root (/). For example, if the myapplication.conf file is placed in this path on the SMW

/var/opt/cray/imps/config/sets/p0/files/simple_sync/common/files/etc/myapplication.conf

then Simple Sync will place myapplication.conf here on all nodes:

/etc/myapplication.conf

Note that the ownership and permissions of files in the config set are preserved in the copies made to nodes.

For more information and use cases about Simple Sync, see Supplemental Information.

Change Kernel Module Parameters Prior to Boot using Modprobe.d Files and Simple Sync

To make a change to DVS kernel module parameters that will persist across boots, edit the local copy of the DVS configuration file (modprobe.d/dvs-local.conf), and then place it under the Simple Sync directory structure that targets the appropriate nodes in the system. The lines to add to modprobe.d/dvs-local.conf are provided for each kernel module parameter in the list that follows. Substitute the appropriate Simple Sync directory structure (that is, the one that targets the appropriate nodes) for <simple sync path> shown in the instructions for each parameter.

Procedure for Changing Kernel Module Parameters Prior to Boot using Modprobe.d Files and Simple Sync

In this example, DVS request logging is enabled on a node with hardware ID c0-0c0s3n2 in config set p0.

Change directory to the Simple Sync root directory.

smw# cd /var/opt/cray/imps/config/sets/p0/files/simple_sync

Create and change to the directory under the Simple Sync root that targets the appropriate nodes. Use the above table to determine what directory that should be. This example targets a particular hardware ID, so the directory is hardwareid/c0-0c0s3n2/files/.
```
smw# mkdir -p hardwareid/c0-0c0s3n2/files
smw# cd hardwareid/c0-0c0s3n2/files
```
The file path that results is referred to as <simple sync path> in the "To change prior to boot" instructions for each kernel module parameter listed below.
Create and change to the directory structure that is to be replicated on the target node. For a modprobe.d file, that is /etc/modprobe.d.
```
smw# mkdir -p etc/modprobe.d
smw# cd etc/modprobe.d
```
Create or edit the DVS configuration file and add the options line(s) for the parameter to be set/changed. For this example, the lines are from the dvs_request_log_enabled entry of the DVS kernel module parameters list below. Then comment/uncomment the appropriate lines, depending on which action is to be taken.
```
# Disable DVS request log
options dvsproc dvs_request_log_enabled=0
```
```
# Enable DVS request log
options dvsproc dvs_request_log_enabled=1
```
When done with all changes to kernel module parameters, proceed to Validate the Config Set and Run Ansible Plays.

Change Kernel Module Parameters Dynamically using Proc Files

Some of the kernel module parameters in the following list can be changed dynamically by echoing values to /proc files on the appropriate nodes. Those that can be changed using that method are indicated in the list, including the name of the /proc file and the values to use. Note that such changes do not persist.

List of DVS Kernel Module Parameters not Accessible through the Configurator

dvs_request_log_enabled

Logs each DVS request sent to servers.

Default value: 1 (enabled)
To view read-only: cat /sys/module/dvsproc/parameters/dvs_request_log_enabled

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Disable DVS request log
options dvsproc dvs_request_log_enabled=0

# Enable DVS request log
options dvsproc dvs_request_log_enabled=1

To change dynamically:

hostname# echo 0 > /proc/fs/dvs/request_log
hostname# echo 1 > /proc/fs/dvs/request_log
hostname# echo 2 > /proc/fs/dvs/request_log

The value 2 resets the log.

dvs_request_log_size_kb

Size (KB) of the request log buffer.

Default value: 16384 KB (16384 * 1024 bytes)
To view read-only: cat /sys/module/dvsproc/parameters/dvs_request_log_size_kb

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set size (in kb) of the request log buffer
options dvsproc dvs_request_log_size_kb=17000

To change dynamically:

hostname# echo 17000 > /proc/fs/dvs/request_log_size_kb

To determine the current buffer size, cat the file. For example:

hostname# cat /proc/fs/dvs/request_log_size_kb
16384

dvs_request_log_min_time_secs

Defines a threshold of time for data to be logged to the request_log.

Default value: 15 seconds
To view read-only: cat /sys/module/dvsproc/parameters/dvs_request_log_time_min_secs

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set threshold (in seconds) for time a DVS request
# takes before logged. Requests taking fewer seconds
# will not be logged.
options dvsproc dvs_request_log_time_min_secs

To change dynamically:

hostname# echo 15 > /proc/fs/dvs/request_log_time_min_secs

dvs_fs_log_enabled

Logs information on I/O operations made from DVS to the underlying file system on DVS server nodes.

Default value: 1 (enabled)
To view read-only: cat /sys/module/dvsproc/parameters/dvs_fs_log_enabled

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Disable DVS fs log
options dvsproc dvs_fs_log_enabled=0

# Enable DVS fs log
options dvsproc dvs_fs_log_enabled=1

To change dynamically:

hostname# echo 0 > /proc/fs/dvs/fs_log
hostname# echo 1 > /proc/fs/dvs/fs_log
hostname# echo 2 > /proc/fs/dvs/fs_log

dvs_fs_log_size_kb

Size (KB) of the log buffer.

Default value: 32768 KB (32768 * 1024 bytes)
To view read-only: cat /sys/module/dvsproc/parameters/dvs_fs_log_size_kb
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:
```
# Set size (in kb) of the fs log buffer
options dvsproc dvs_fs_log_size_kb=17000
```

To change dynamically:

hostname# echo 17000 > /proc/fs/dvs/fs_log_size_kb

To determine the current buffer size, cat the file. For example:

hostname# cat /proc/fs/dvs/fs_log_size_kb
32768

dvs_fs_log_min_time_secs

Defines a threshold of time for data to be logged to the fs_log.

Default value: 15 seconds
To view read-only: cat /sys/module/dvsproc/parameters/dvs_fs_log_time_min_secs

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set threshold (in seconds) for time a DVS requests
# takes before logged. Requests taking fewer seconds
# will not be logged.
options dvsproc dvs_fs_log_time_min_secs

To change dynamically:

hostname# echo 15 > /proc/fs/dvs/fs_log_time_min_secs

dvs_instance_info

Contains the following fields, which are parameters for the DVS thread pool in the DVS IPC layer. Most of these fields can be changed through other module parameters (e.g., dvsipc_msg_thread_limit and dvsipc_single_msg_queue); however, this module parameter has priority over the individual ones and if set, will override them.


Field	Definition
thread_min	Number of threads created at startup.
thread_max	Maximum number of persistent threads.
thread_limit	Maximum number of valid threads that DVS IPC allows to exist at one time. DVS IPC will dynamically scale up the number of threads to this number as the load increases.
thread_concurrent_creates	Maximum number of IPC threads that can be in the process of forking a new thread.
thread_nice	Nice value for the threads.
single_msg_queue	Disables/enables fairness of service. Setting to 1 disables fairness of service by processing incoming requests with a FIFO method. Setting to 0 (default) groups requests into different queues (qhdrs) based on apid, and round-robbins the queues to maintain quality of service (QOS) among jobs.
init_free_qhdrs	Low water mark for the pool of unused qhdrs. When the pool falls below this number, more are allocated. Used only if single_msg_queue = 0.
max_free_qhdrs	Maximum number of unused qhdrs that can exist before the system starts freeing them. Used only if single_msg_queue = 0.
Interactions among fields: thread_min <= thread_max <= thread_limit thread_concurrent_creates <= thread_limit init_free_qhdrs and max_free_qhdrs used only when single_msg_queue == 0

Important: For this parameter to be valid, values must be specified for all of the fields. To avoid unintentional changes, be careful when changing any field values.

Default value: see modprobe.d examples
To view read-only: cat /sys/module/dvsipc/parameters/dvs_instance_info
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf (comment out either the compute or service line, depending on the type of node(s) being configured):
```
# Set parameters for DVS thread pool in DVS IPC layer
# Defaults for compute nodes
options dvsipc dvs_instance_info=4,16,1000,0,0,0,1,1

# Defaults for service nodes
options dvsipc dvs_instance_info=16,64,1000,0,0,0,64,2048
```
For compute nodes, this translates to dvs_instance_info = thread_min = 4, thread_max = 16, thread_limit = 1000, thread_concurrent_creates = 0, thread_nice = 0, single_msg_queue = 0, init_free_qhdrs = 1, max_free_qhdrs = 1.
For service nodes, this translates to dvs_instance_info=thread_min = 16, thread_max = 64, thread_limit = 1000, thread_concurrent_creates = 0, nice = 0, single_msg_queue = 0, init_free_qhdrs = 64, max_free_qhdrs = 2048.
Note that if using the defaults for a compute node, ensure that dvsipc_config_type=0 is also set, and likewise, for a service node, ensure that dvsipc_config_type=1 for consistency.
To change dynamically: N/A

dvsipc_config_type

Forces DVS to load in a mode optimized for DVS clients (0) or servers (1). This parameter can be used to make DVS load in a non-default manner. Frequently used for repurposed compute nodes.

Default value: 0 for compute nodes, 1 for service nodes
To view read-only: cat /sys/module/dvsipc/parameters/dvsipc_config_type
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf (comment out either the client or server line, depending on the type of node(s) being configured):
```
# Load DVS as a client
options dvsipc dvsipc_config_type=0

# Load DVS as a server
options dvsipc dvsipc_config_type=1
```

To change dynamically:

hostname# echo 0 > /proc/fs/dvs/ipc/config-type
hostname# echo 1 > /proc/fs/dvs/ipc/config-type

dvsipc_single_msg_queue

Disables fairness of service, which is enabled by default. Setting this parameter to 1 disables fairness of service by forcing DVS to use a single message queue instead of a list of queues.

Default value: 0 (fairness of service enabled)
To view read-only: cat /sys/module/dvsipc/parameters/dvsipc_single_msg_queue
To change prior to boot, use the single_msg_queue field of the dvs_instance_info parameter. If no other fields in dvs_instance_info need to be changed, it may be easier to change the dvsipc_single_msg_queue parameter directly by adding these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:
```
# Disable fairness of service
options dvsipc dvsipc_single_msg_queue=1

# Enable fairness of service
options dvsipc dvsipc_single_msg_queue=0
```
To change dynamically: N/A

dvsof_concurrent_reads

Controls how many threads are allowed into the read path on the server. A value of -1 disables, 0 uses the number of cores on the CPU, and any other positive number sets the number of threads. Set to 0 for best DataWarp performance.

Default value: -1
To view read-only: cat /sys/module/dvs/parameters/dvsof_concurrent_reads

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Disable concurrent reads
options dvs dvsof_concurrent_reads=-1

# Set number of threads able to do concurrent 
# reads = number of cores on CPU
options dvs dvsof_concurrent_reads=0

# Set number of threads able to do concurrent 
# reads = a positive number (e.g., 3)
options dvs dvsof_concurrent_reads=3

To change dynamically: N/A

dvsof_concurrent_writes

Controls how many threads are allowed into the write path on the server. A value of -1 disables, 0 uses the number of cores on the CPU, and any other positive number sets the number of threads. Set to 0 for best DataWarp performance.

Default value: -1
To view read-only: cat /sys/module/dvs/parameters/dvsof_concurrent_writes

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Disable concurrent writes
options dvs dvsof_concurrent_writes=-1

# Set number of threads able to do concurrent 
# writes = number of cores on CPU
options dvs dvsof_concurrent_writes=0

# Set number of threads able to do concurrent 
# writes = a positive number (e.g., 3)
options dvs dvsof_concurrent_writes=3

To change dynamically: N/A

dvsproc_stat_control

(deprecated) Controls DVS statistics. This legacy parameter has been maintained for backward compatibility, but values are overridden by dvsproc_stat_defaults, if specified.

Default value: 1 (enabled)
To view: cat /sys/module/dvsproc/parameters/dvsproc_stat_control

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Disable DVS statistics
options dvsproc dvsproc_stat_control=0

# Enable DVS statistics
options dvsproc dvsproc_stat_control=1

To change dynamically:
This is root writable at /sys/module/dvsproc/parameters/dvsproc_stat_control, but changes should be made only through the /proc/fs/dvs/stats interface, as shown in this example.
```
hostname# echo 0 > /proc/fs/dvs/stats
hostname# echo 1 > /proc/fs/dvs/stats
hostname# echo 2 > /proc/fs/dvs/stats
```

dvsproc_stat_defaults

Controls DVS statistics. Use this parameter to disable/enable and format DVS statistics.

Default values: enable, legacy, brief, plain, notest
To view: cat /sys/module/dvsproc/parameters/dvsproc_stat_defaults

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Disable/enable and format DVS statistics
options dvsproc dvsproc_stat_defaults="enable,legacy,brief,plain,notest"

To change dynamically:
This is root writable at /sys/module/dvsproc/parameters/dvsproc_stat_defaults, but changes should be made only through the /proc/fs/dvs/stats interface, as shown in this example.
```
hostname# echo disable > /proc/fs/dvs/stats
hostname# echo enable > /proc/fs/dvs/stats
hostname# echo reset > /proc/fs/dvs/stats
hostname# echo json,pretty > /proc/fs/dvs/stats
```

estale_max_retry

Controls the number of times to retry an operation on the original server after it returns ESTALE.

Default value: 36 iterations at a fixed 5 seconds per iteration (3 minutes)
To view read-only: cat /sys/module/dvsproc/parameters/estale_max_retry

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set the number of times to retry an operation 
# on the original server after it returns ESTALE
options dvsproc estale_max_retry=

To change dynamically (example changes estale_max_retry to 40 for illustration only):
```
hostname# echo 40 > /proc/fs/dvs/estale_timeout_secs
```

estale_timeout_secs

Controls the time to wait between retries of an operation after it returns ESTALE.

Default value: 300 seconds
To view read-only: cat /sys/module/dvsproc/parameters/estale_timeout_secs

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set the time to wait between retries of an 
# operation that returns ESTALE
options dvsproc estale_timeout_secs=

To change dynamically (example changes estale_timeout_secs to 400 for illustration only):
```
hostname# echo 400 > /proc/fs/dvs/estale_timeout_secs
```

kdwfs_instance_info

Contains the following fields, which are parameters for the DataWarp thread pool in the DVS IPC layer.


Field	Definition
thread_min	Number of threads created at startup.
thread_max	Maximum number of persistent threads.
thread_limit	Maximum number of valid threads that DVS IPC allows to exist at one time. DVS IPC will dynamically scale up the number of threads to this number as the load increases.
thread_concurrent_creates	Maximum number of IPC threads that can be in the process of forking a new thread.
thread_nice	Nice value for the threads.
single_msg_queue	Disables/enables fairness of service. Setting to 1 disables fairness of service by processing incoming requests with a FIFO method. Setting to 0 (default) groups requests into different queues (qhdrs) based on apid, and round-robbins the queues to maintain quality of service (QOS) among jobs.
init_free_qhdrs	Low water mark for the pool of unused qhdrs. When the pool falls below this number, more are allocated. Used only if single_msg_queue = 0 .
max_free_qhdrs	Maximum number of unused qhdrs that can exist before the system starts freeing them. Used only if single_msg_queue = 0.
Interactions among fields: thread_min <= thread_max <= thread_limit (set all three equal for best DataWarp performance) thread_concurrent_creates <= thread_limit init_free_qhdrs and max_free_qhdrs used only when single_msg_queue == 0

Important: For this parameter to be valid, values must be specified for all of the fields. To avoid unintentional changes, be careful when changing any field values.

Default value: 1,1,1024,4,-10,1,1,1
To view read-only: cat /sys/module/dvsipc/parameters/kdwfs_instance_info
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-dws.conf (values shown are defaults):
```
# Set parameters for DataWarp thread pool in DVS IPC layer
options dvsipc kdwfs_instance_info=256,256,1024,4,-10,1,1,1
```
This translates to kdwfs_instance_info thread_min = 256, thread_max = 256, thread_limit = 1024, thread_concurrent_creates = 4, nice = -10, single_msg_queue = 1, init_free_qhdrs = 1, max_free_qhdrs = 1.
To change dynamically: N/A

lnd_name

lnd_name uniquely identifies the LNet network that DVS will use. DVS communicates it to the LNet service when DVS is being initialized. It must match the cray_lnet.settings.local_lnet.data.lnet_name value set in the cray_lnet service for DVS to boot properly. To find that value, search the CLE config set (this example searches in config set p0 and finds lnet_name = gni4):

smw# cfgset search --term lnet_name \
--state all --service cray_lnet p0
# 1 match for 'lnet_name' from cray_lnet_config.yaml
#----------------------------------------------------
cray_lnet.settings.local_lnet.data.lnet_name: gni4

Default value: gni99
To view read-only: not visible in /sys/module
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf, substituting for gnix the value found from the config set search:
```
# Set identifier of LNet network DVS will use
options dvsipc_lnet lnd_name=gnix
```
To change dynamically: N/A

sync_dirty_timeout_secs

On DVS servers, specifies the number of seconds that must have passed since the file was written before DVS syncs it. The objective is to reduce unnecessary sync operations for files actively being updated. Decreasing this number increases the likelihood that the file is in use when it is synced. Increasing this number increases the likelihood that processes are killed during a server failure.

On DVS clients, specifies the number of seconds that must have passed since the file was written before DVS asks the server for an updated sync time. Decreasing this number increases the number of DVS requests being sent. Increasing this number increases the likelihood that processes are killed during a server failure.

Default value: 300 (servers and clients)
To view read-only: cat /sys/module/dvs/parameters/sync_dirty_timeout_secs

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set the timeout (seconds) for syncing 
# dirty files on a DVS server or client
options dvs sync_dirty_timeout_secs=300

To change dynamically:

hostname# echo 300 > /proc/fs/dvs/sync_dirty_timeout_secs

sync_num_threads

Specifies the number of threads on the DVS server that perform sync operations. The number of threads must be a minimum of 1 thread and a maximum of 32. Note that it can take up to sync_period_secs for changes to this value to take effect.

Default value: 8
To view read-only: cat /sys/module/dvs/parameters/sync_num_threads

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set the number of threads that perform 
# sync operations on a DVS server
options dvs sync_num_threads=8

To change dynamically:

hostname# echo 8 > /proc/fs/dvs/sync_num_threads

sync_period_secs

On DVS servers, specifies the number of seconds before the sync_num_threads syncs files on the DVS server (if necessary).

On DVS clients, specifies the number of seconds between checks for dirty files that need to request the last sync time from the server.

Default value: 300 (server), 600 (client)
To view read-only: cat /sys/module/dvs/parameters/sync_period_secs
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:
```
# Set the sync period (seconds) on DVS server/client
options dvs sync_period_secs=300
```

To change dynamically:

hostname# echo 300 > /proc/fs/dvs/sync_period_secs