Periodic Sync Promotes Data and Application Resiliency

DVS periodic sync improves data and application resiliency and is more efficient than closesync.

DVS periodic sync improves data resiliency and facilitates a degree of application resiliency so that applications may continue executing in the event of a stalled file system or DVS server failure. Without periodic sync, such an event would result in DVS clients killing any processes with open files that were written through the failed server. Any data written through that server that was only in the server's page cache and not written to disk would be lost, and processes using the file would see data corruption.

Periodic sync works by periodically performing an fsync on individual files with written data on the DVS servers, to ensure that those files are written to disk. For each file, the DVS client tracks when a DVS server performs a file sync and when processes on DVS clients write to it, and then notifies the DVS server when fsync is needed. Periodic sync functions similarly to the DVS closesync mount option, but it is more efficient because it is aware of which files may have written data. Unlike closesync and datasync, periodic sync can sync data over time asyncronously so that the client does not need to wait for the sync operation to complete.

DVS periodic sync is effectively enabled by default because the sync_period_secs parameter, which affects the amount of time between syncs on the server, is non-zero by default. The only way to effectively disable periodic sync is to set that parameter to 0. Cray recommends keeping periodic sync enabled instead of using the closesync mount option.

Use the following three parameters to tune DVS periodic sync behavior. Because these parameters cannot be set using the configurator, use the procedure Change Kernel Module Parameters Prior to Boot using Modprobe.d Files and Simple Sync and the following information, both from the topic Configure DVS using Modprobe or Proc Files.

sync_num_threads

Specifies the number of threads on the DVS server that perform sync operations. The number of threads must be a minimum of 1 thread and a maximum of 32. Note that it can take up to sync_period_secs for changes to this value to take effect.

Default value: 8
To view read-only: cat /sys/module/dvs/parameters/sync_num_threads

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set the number of threads that perform 
# sync operations on a DVS server
options dvs sync_num_threads=8

To change dynamically:

hostname# echo 8 > /proc/fs/dvs/sync_num_threads

sync_dirty_timeout_secs

On DVS servers, specifies the number of seconds that must have passed since the file was written before DVS syncs it. The objective is to reduce unnecessary sync operations for files actively being updated. Decreasing this number increases the likelihood that the file is in use when it is synced. Increasing this number increases the likelihood that processes are killed during a server failure.

On DVS clients, specifies the number of seconds that must have passed since the file was written before DVS asks the server for an updated sync time. Decreasing this number increases the number of DVS requests being sent. Increasing this number increases the likelihood that processes are killed during a server failure.

Default value: 300 (servers and clients)
To view read-only: cat /sys/module/dvs/parameters/sync_dirty_timeout_secs

To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:

# Set the timeout (seconds) for syncing 
# dirty files on a DVS server or client
options dvs sync_dirty_timeout_secs=300

To change dynamically:

hostname# echo 300 > /proc/fs/dvs/sync_dirty_timeout_secs

sync_period_secs

On DVS servers, specifies the number of seconds before the sync_num_threads syncs files on the DVS server (if necessary).

On DVS clients, specifies the number of seconds between checks for dirty files that need to request the last sync time from the server.

Default value: 300 (server), 600 (client)
To view read-only: cat /sys/module/dvs/parameters/sync_period_secs
To change prior to boot, add these lines to <simple sync path>/etc/modprobe.d/dvs-local.conf:
```
# Set the sync period (seconds) on DVS server/client
options dvs sync_period_secs=300
```

To change dynamically:

hostname# echo 300 > /proc/fs/dvs/sync_period_secs

A fourth /proc file, /procsfs/dvs/sync_stats, collects statistics of the syncing behavior. This setting is not tunable (read only).