Run Tests after Boot is Complete

Tests are performed as crayadm from the login node and include running apstat to get number of nodes, running aprun on all nodes, checking the home directory, the Lustre directory, and the SMW databases to ensure they have the current state of all compute nodes.

This procedure assumes the following:
  • The system has completed booting.
  • The compute nodes are “interactive" (i.e., not under workload manager control).
  • ALPS is available.

If ALPS is not available and Slurm is used as the workload manager (WLM), then the compute nodes can be either "interactive" or "batch" and srun (the Slurm command equivalent to aprun) should be used instead of the aprun commands in the steps that follow.

Log in to the login node as crayadm. This can be done from the SMW to the boot node to the login node, or directly from another computer to the login node without passing through the SMW and boot node. Then perform these rudimentary functionality checks.

  1. Run apstat to get the number of nodes to use for the following commands.
    crayadm@login> NUMNODES=$(apstat | awk '$1=="XT" {print $3}')
crayadm@login> echo NUMNODES is $NUMNODES
  2. Verify that all nodes run (from /tmp).
    crayadm@login> cd /tmp
crayadm@login> aprun -b -n $NUMNODES -N 1 /bin/cat /proc/sys/kernel/hostname
  3. Verify that the home directory is working by running a job.
    crayadm@login> cd ~
crayadm@login> aprun -b -n $NUMNODES -N 1 /bin/cat /proc/sys/kernel/hostname
  4. Verify that the Lustre directory is working by running a job.
    crayadm@login> cd /lustre_file_system
crayadm@login> aprun -b -n $NUMNODES -N 1 /bin/cat /proc/sys/kernel/hostname