The gpustat Data Plugin

Collect NVIDIA GPU statistics.

The gpustat plugin collects the following utilization statistics for NVIDIA GPUs, if present. The data is written in JSON list format.

gpuids: Total number of processes using GPUs
gpusecs: Time spent processing on GPUs
maxgpusecs: Maximum GPU time across all nodes
maxmem: Maximum memory used across all nodes
summem: Total memory used across all nodes

RUR `gpustat` output

This example shows gpustat data as written in /var/opt/cray/log/partition-current/messages-date on the SMW.

2019-06-07T15:50:42.761257-05:00 c0-0c0s2n2 RUR 11329 p2-20130709t145714 [RUR@34] uid: 12345, apid: 841052, jobid: 154239, cmdname: /tmp/dostuff plugin: gpustats ['maxmem', 10363864, 'summem', 39102708, 'gpusecs', 162610, 'maxgpusecs', 81305, 'gpuids', 2]