Retrying Failed Operation

Volume-level

If an offload, recall, or abort operation for a volume fails, the volume tierjobstatus command shows one of the following statuses:

FailureFatal	Indicates failure is fatal and CLDB cannot retry the operation again.
FailureRetriable	Indicates failure to offload; however, CLDB will try again if the job is not manually restarted again or aborted.

CLDB tries the operation again (up to 5 times by default) after a specific wait time (of 30 minutes by default) for the following errors:

• EAGAIN
• ETIMEDOUT
• ENETUNREACH
• ENETDOWN
• ECONNRESET

The RetryCount field value in the volume tierjobstatus command output shows the number of times CLDB has retried so far. For example:

# maprcli volume tierjobstatus -name testvol -json
{
 "timestamp":1503308792266,
 "timeofday":"2017-08-21 09:46:32.266 GMT+0000",
 "status":"OK",
 "total":1,
 "data":[
  {
   "offload":{
    "state":"FailureRetriable, RetryCount: 5",
    "startTime":"2017-08-21 09:07:17.506 GMT+0000",
    "endTime":"2017-08-21 09:08:49.799 GMT+0000",
    "gateway":"10.10.102.68:8660"
   }
  }
 ]
}

File-level

If the offload or recall operation for an individual file fails, the file tierstatus or hadoop mfs command returns one of the following:

Code	Message	Description
0	HAS_LOCAL_DATA	Indicates that the file is not yet fully offloaded.
1	NO_LOCAL_DATA	Indicates that the file was completely offloaded.
2	OP_FAIL	Indicates that the operation to retrieve the status failed.
3	INVALID_FILE	Indicates that the file does not exist.
4	FILE_NOT_TIERED	Indicates that the file is not in a tiered volume.
5	FILE_EMPTY	Indicates that the file specified for offload is an empty file.
6	NO_GATEWAY	Indicates that no MAST Gateways are available for offload operation.
7	OP_TIMEOUT	Indicates that there was no response from the MAST Gateway (maybe as a result of an error) during the offload or recall operation.
8	FTOS_SUCCESS	Indicates that the file was successfully offloaded or recalled.
9	FTOS_ABORTED	Indicates that the file offload or recall operation was aborted.
10	FTOS_ABORT_IN_PROGRESS	Indicates that the file offload or recall job is being aborted.
11	FTOS_TRANSFER_IN_PROGRESS	Indicates that the file offload is in progress.
12	FTOS_REQ_QUEUED	Indicates that the file offload is scheduled, but has not yet started.
13	FTOS_JOB_NOT_AVAILABLE	Indicates that the job ID specified with the tierjobstatus command is not available.

When a file-level offload or recall operation fails, CLDB does not retry the operation. For failed file-level:

• Offload operation, you can run the command to retry the operation. For more information, see Offloading a File to a Tier Using the CLI and REST API. Alternatively, if the volume that the file is associated with has a data offload schedule, the file data is automatically offloaded based on the rules associated with the volume.
• Recall or abort operation, you can run the command again to retry the operation if the error returned is not EIO.

You can configure the number of times CLDB retries and the interval between retries using the CLI.