Data Reads, Writes, and Recalls

Read of Tiered Data

Depending on whether the standard volume data is outside the data-fabric cluster and in the cloud (cold tiering) or on the data-fabric cluster (warm tiering), data-fabric processes the request to read standard volume data and mirror volume data.

Data Reads on Tiering-Enabled Standard Volumes

Data Fabric processes client requests to read standard volume data on the warm tier and the cold tier differently.

Warm Tier: When a client attempts to read, the read request is first sent to the front-end volume and if the data exists in the front-end volume, the data is returned from the front-end volume. If data is not in the front-end volume, the data is returned from the erasure coded volume.
Cold Tier: When a client attempts to read, the read request is first sent to the volume on the data-fabric cluster and if the data exists in the volume on the cluster, the data is returned from the volume. On the other hand, if the data was offloaded, the MAST Gateway recalls the data from the cold-tier to process the read request. See Recall of Tiered Data for more information on recalled data.

The following illustration shows a client sending the data read request first (#1) to the tiering-enabled volume and the response (#2) being served from the volume on the data-fabric cluster. Then (#3), data is offloaded to the back-end erasure coded volume (for Volume A) and to the cloud (for Volume B). When the client next sends a read request to the volume on the data-fabric cluster (#4), for:

Volume A, the MAST Gateway forwards the request to the back-end erasure-coded volume (#5) from where data is returned (#6) to the client.
Volume B, the MAST Gateway recalls the data (#5 and #6) from the cloud to the volume on the data-fabriccluster, from where data is returned (#7) to the client.

Data Reads on Tiering-Enabled Mirror Volumes

Data Fabric processes client requests to read mirror volume data on the warm tier and the cold tier differently.

Warm Tier: When a client attempts to read, the read request is first sent to the front-end volume and if the data exists in the front-end mirror volume, the data is returned from the front-end volume. If data is not in the front-end volume, the data is returned from the erasure coded volume.
Cold Tier: When a client attempts to read, the read request is first sent to the volume on the data-fabric cluster and if the data exists in the volume on the cluster, the data is returned from the volume. On the other hand, if the data was offloaded, the MAST Gateway recalls or fetches a copy of the data (from the tier) into an associated cache-volume, from where data is returned to the client. See Recall of Tiered Data for more information on recalled data.

The following illustration shows a client sending the read request first (#1) to the tiering-enabled mirror volume and the response (#2) being served from the volume on the data-fabric cluster. Then (#3), data is offloaded to the back-end erasure coded volume (for warm tier) and to the cloud (for cold tier). When the client next sends a read request to the volume on the data-fabric cluster (#4), for:

Warm tier, data is returned from the back end erasure-coded volume (#5).
Cold tier, data is recalled in the associated cache-volume (#5 and #6), from where data is returned to the client (#7).

The cache-volume named mapr.internal.cv.<volume-name>_<id> is created when the tiering-enabled mirror volume is created. Although it may not hold any data initially, a copy of the tiered data is fetched into the cache-volume whenever there is a read of the cold-tiered mirror volume data or explicit recall of (cold or warm) tiered mirror volume data. You can use the volume info command on the tiering-enabled mirror volume to get the offload and recall statistics, which are from the cache-volume, for the mirror volume.

The cache-volume has the same replication factor as the mirror volume (at the time of volume creation); changes to the mirror volume replication factor do not trigger a change to the replication factor of the associated cache-volume.

If the tiering-enabled mirror volume is deleted, the cache-volume is also deleted. If the tiering-enabled mirror volume is promoted to a read-write volume, the associated cache-volume is deleted.

Write on Tiered Data

When writes happen, if the write is:

An append, new data is offloaded when the data meets the criteria in the rule (associated with the volume) for offload.
A change to existing data (overwrite), the data is recalled to the data-fabric filesystem to allow the write to succeed and then offloaded when the data meets the criteria in the rule (associated with the volume) for offload. See Recall of Tiered Data for more information on recalled data.

Note: If cold data is accessed (read/written) frequently, I/O to that file may suffer large latencies. In such scenarios, recall the whole volume or the corresponding files.

Recall of Tiered Data

Offloaded data is automatically recalled when a client performs a read or overwrite on the data in the cold-tier, or when a client performs an overwrite on the data in the warm-tier. The MAST Gateway fetches a copy of the data to allow the operations to succeed. You can also manually trigger a recall of:

All volume data using the maprcli command or REST API.
See Recalling a Volume to filesystem for more information.
File using the hadoop command, maprcli command, REST API, (loopbacknfs or FUSE-based) POSIX client, or the NFS client.
See Recalling a File to filesystem Using the CLI and REST API and Running Tiering Commands when maprcli and hadoop Commands are not Available for more information.

Based on the expiration time period set at the volume level for recalled data, recalled data is:

Offloaded again based on the rules if there are changes to the data.
Purged when the compactor runs if there are no changes to the data.

For a cold tiering volume, explicitly recall the volume before running any analytics jobs.

For mirror volumes, when you recall tiered data, data from the tier is recalled into an associated cache-volume, which is created at the time of the creation of the tiering-enabled mirror volume. For all explicit recall of warm-tiered data and explicit and automatic recalls of cold-tiered data, the MAST Gateway recalls data into the associated cache-volume. The data in the cache-volume is "hot" in the cluster, or available for reads, for the duration of the expiry-period. The recalled data is purged by the compactor when the expiration time that is set at the volume level is reached or has passed.

If the recall fails, CLDB retries the operation after some time. See Retrying Failed Operation for more information.