Quiesce a DVS-projected File System

Describes how quiesce works and provides several use cases.

Sites can use the DVS quiesce capability to temporarily suspend traffic to a DVS-projected file system on any number of DVS servers. When a directory is quiesced, the DVS server completes any outstanding requests but does not honor any new requests for that directory. When the quiesce is finished, administrators know it is safe to perform operations on the quiesced directory without any possibility of interference by a DVS client. DVS quiesce can be used when a file system needs to be repaired or to safely take a DVS server node out of service.

CAUTION: Because it may cause data corruption, do not use DVS quiesce to:

Quiesce a directory that is being used by DataWarp
Quiesce a directory on every DVS server

To use DVS quiesce, an administrator writes into /proc/fs/dvs/quiesce, as shown in the following use cases.

Use Case 1: Quiesce a single directory on a single DVS server

An admin wants to quiesce a directory /gpfs/test/foo on a DVS server. This is an unlikely use case but an illustrative example.

On the DVS server, quiesce the directory.
```
dvs1# echo quiesce /gpfs/test/foo > /proc/fs/dvs/quiesce
```
When the directory quiesce has completed, the shell prompt will return to user control. There is no command output.

When finished with the directory, unquiesce it.

dvs1# echo unquiesce /gpfs/test/foo > /proc/fs/dvs/quiesce

Ensure that the directory is no longer on the quiesced list.
```
dvs1# cat /proc/fs/dvs/quiesce
```

Use Case 2: Quiesce all directories on a DVS server

An admin wants to remove a DVS server from service but wants to let any outstanding request complete first.

On the DVS server, quiesce all directories on that server.
```
dvs1# echo quiesce / > /proc/fs/dvs/quiesce
```
When the quiesce has completed, the shell prompt will return to user control. There is no command output.
To allow traffic to this server to resume, unquiesce all of its projected directories.
```
dvs1# echo unquiesce / > /proc/fs/dvs/quiesce
```

How Quiesce Works: the Userspace Application View

Userspace applications have no visibility into any specific quiesce information. A quiesced directory will present in one of two ways:

Entirely normally, if the directory is quiesced only on servers that the application is not using.
Useable but with degraded performance, if the application finds that its main server is quiesced and must query other servers.

How Quiesce Works: the Server View

To provide the quiesce capability, DVS servers keep a list of quiesced directories and the current outstanding requests on each quiesced directory. When an admin requests that DVS quiesce a directory on a server, DVS does the following:

Adds that directory to the server's list of quiesced directories
Iterates over all open files, closing any file that resides in the quiesced directory and setting a flag to indicate that the file was closed due to the directory being quiesced

When a DVS server receives a request from a client, DVS checks the request path against the list of quiesced directories. The comparison between the path name in the request and the quiesced directory is a simple string compare to avoid any access of the underlying file system that has been quiesced. If DVS finds that the request is for a quiesced file system, it sends a reply indicating that the request could not be completed due to a quiesce and noting which directory is quiesced. If the client request is for a file that has been closed due to quiesce, the server returns a reply to the client indicating that the request could not be completed due to a quiesce.

When an admin unquiesces a directory on a DVS server, DVS simply removes that directory from the server's list of quiesced directories and clears all quiesce-related flags for that directory.

How Quiesce Works: the Client View

When making a request of a server, a client may get back reply indicating that the request was for a file in a quiesced directory. The client then retries the operation on the next server in its server list. If it makes the request of every server in its server list and gets the same reply from each of them, then one of two things happens, depending on the type of request:

path name request: If the request is a path name request (lookup, stat, file open, etc.), then DVS reattempts the operation on a different server in a round-robin fashion until it finds a server that allows the operation to complete successfully.
open file: If the request is for an open file (read, write, lseek, etc.), then DVS attempts the operation on a different server. If the file is not open on any other servers, DVS attempts to open on the file on a server in a round robin fashion until it gets a successful open. DVS will then attempt to perform the operation.

If a client receives a reply indicating a quiesced directory, the client adds that directory to a list of quiesced directories held on the DVS superblock. This is intended to reduce network traffic by avoiding requests that target quiesced directories. The client's list of quiesced directories expires about every 60 seconds, thereby allowing clients to try those directories again in case one or more have been unquiesced during that time. This mechanism enables DVS to strike a balance between the timely unquiescing of a file system and a large reduction in network traffic and requests coming into the server. It also has the effect of naturally staggering clients when they start to use a server.