Annotations

Describes the View for ClusterStor feature for recording and retrieving important events that happen in either the InfiniBand network or the monitored ClusterStor systems

The View for ClusterStor™ server includes an annotation facility for recording and retrieving important events that happen in either the InfiniBand network or the monitored ClusterStor systems. These events are recorded to Elasticsearch, much like logs. A number of these events in the system log fields are displayed in View for ClusterStor as annotations or are highlighted with warning icons in the System Overview tiles.

There are currently three classifications of event types:
  • Lustre - Critical Lustre errors noted in the ClusterStor logs.
  • Health - SNMP hardware events noted from a polling of SNMP.
  • InfiniBand - Changes in InfiniBand topology.

The following figure shows an example of annotations on the Lustre File System Metadata Dashboard:

Lustre Event Type

View for ClusterStor highlights when Lustre has issued a critical error on the ClusterStor system. The critical Lustre alerts will be written to the Elasticsearch snx-logs_* indices. These are displayed in the graphs as Lustre annotations.

InfiniBand Event Type

View for ClusterStor tracks and issues alerts when changes to the IB topology occur. It compares the current topology to a saved baseline configuration every 900 seconds. The baseline is the normal operating IB topology.

When View for ClusterStor detects an IB topology transition to or from the baseline, it will send an alarm notification email and log the error in Elasticsearch. The ClusterStor system service alerts will be written to the Elasticsearch ~event-ibtopology_* indices. These are displayed in the graphs as InfiniBand annotations.

Health Event Type

Health events are a collection of zero or greater service alerts that are present in the ClusterStor system storage cluster. View for ClusterStor gathers health events by parsing through ClusterStor system log files. The composition of the ClusterStor log files varies depending on the software release that is running on the ClusterStor system. As a result, the health events that appear as annotations in View for ClusterStor are heavily dependent on the ClusterStor release. For example, the following table illustrates the success in detecting OST and MDT failover/failback events across 3 different ClusterStor system updates (SU):
ClusterStor ReleaseOST Failover/BackMDT Failover/Back
3.0.0 SU-011HighHigh
3.0.0 SU-010HighLow
2.0.0 SU-028MediumLow

Please note that View for ClusterStor annotations are intended to provide system administrators with insight into the ClusterStor system's performance characteristics that are being displayed in Grafana dashboards, but are not intended to provide a definitive picture.

The ClusterStor system service alerts are written to the Elasticsearch event-* indices and are displayed in the graphs as Health annotations.

Some examples of these events may include:
  • FRU Fan Status (event-hdwefan_*)
  • Raid Resync start/complete (event-mdraid_*)
  • Slow drive events (event-hdslowdrive_*)
  • Indications of ClusterStor OST/MDT failover (event-stgfailover_*)
    • OST failover/back event example:

    • MDT failover/back event example: