Editing the Retention Settings of Aggregated Logs

By default, aggregate logs are stored on the MapRHPE Ezmeral Data Fabric filesystem for 30 days. The retention time for aggregated logs also applies to centralized logs.

To edit the retention settings, add or edit the following properties in the yarn-site.xml file:

Set the value of yarn.log-aggregation.retain-seconds to set the duration that the logs are maintained. If you set a negative value for yarn.log-aggregation.retain-seconds, logs will not be deleted.

Note: The duration specified by yarn.log-aggregation.retain-seconds starts from the time that the application starts running. Therefore, when you configure the duration, consider how long you want the log to remain in addition to the amount of time that the application will take to run. For example, if you expect most applications to take 20 seconds to run, do not set the value of this property to 20 seconds because the log might be deleted as soon as the applications completes.
Optionally, set the yarn.log-aggregation.retain-check-interval-seconds to specify how often the log retention check should be run. By default, it is one-tenth of the log retention time.

For more details about the properties that impact the YARN container logs and the aggregation option, see yarn-site.xml.