LMT Overview

Lustre monitoring tool (LMT) for direct-attached Lustre (DAL) on Cray Linux environment (CLE) systems.

The Lustre monitoring tool (LMT) monitors Lustre file system servers metadata target (MDT), object storage target (OST), and Lustre networking (LNet) routers. It collects data using the Cerebro monitoring system and stores it in a MySQL database. Graphical and text clients are provided which display historical and real time data pulled from the database.

There is currently no support for multiple MDTs in the same filesystem (DNE1).

Figure: LMT Block Diagram

View and Aggregate Data

Two commands display data provided by LMT:
  • ltop displays live data
  • lmtsh displays historical data from the MySQL database
Configuration of the data aggregation cron job is enabled by using the cfgset command. 
smw# cfgset update -s cray_lmt -m interactive partition

Interfaces

An LMT MySQL database is accessed using a MySQL client. The database created is named filesystem_fsname where fsname is the name of the filesystem which LMT monitors.

Additional command-line interfaces (CLIs) to LMT are ltop, lmtinit, and lmtsh. These interfaces are only available on the LMT server and lmtinit and lmtsh can only be used by root.
  • ltop provides access to live data collected by LMT
  • lmtinit sets up a MySQL database for LMT
  • lmtsh provides access to data in the LMT MySQL database

The LMT graphical user interface (GUI) package provides two other interfaces to LMT called lwatch and lstat. lwatch is the GUI, and lstat provides output similar to the output of ltop. Any user with network connectivity to the LMT server and credentials for a MySQL account with read access to the LMT database can use the CLI.

LMT also provides data aggregation scripts that act on raw data in the MySQL database and calculate hourly, daily, and monthly data. The main script for aggregation is /usr/share/lmt/cron/lmt_agg.cron.

Dependencies

The MySQL-server runs on the MGS node. The IMPS handles dependencies as long as the packages needed are in the CentOS image repository.

The two-disk RAID which is currently used as the management target (MGT) must split into two volumes in SANtricity. The MGT volume must be 1GB in size. The other volume must be an ext3 volume using the rest of the space on the disks (599GB unformatted).

The LMT GUI requires the Java runtime environment (JRE) and works best with IBM JRE. This is available on the CentOS media for IMPS DAL.

Failover

The failover MGS can be used as the LMT server as long as all LMT agents (Lustre servers) are configured to send Cerebro messages to both the primary and the failover MGSs. There, Cerebro daemon (cerebrod) will be running on the MGS and its failover partner all the time since its failover partner is the metadata server (MDS). However, listening on the failover MGS (the MDS) can be turned off until the MGS failover occurs. The disks used for the MySQL database must be accessible to the primary and failover MGS. The nodes must be prevented from accessing the disks at the same time using STONITH.

If any object storage server (OSS) or MDS fails over, start cerebrod on its failover partner when failover has completed.