Reconfigure SSD-endowed Compute Nodes Dynamically

This procedure shows how to dynamically reconfigure SSD-endowed compute nodes (e.g., KNL nodes, which have Intel® Xeon Phi™ "Knights Landing" processors) using the capmc utility. When rebooted, the hardware inventory on the SDB node is updated.

This procedure assumes that the XC system is booted.

Nodes endowed with Intel® Xeon Phi™ "Knights Landing" processors (KNL nodes) are bootable into a number of NUMA (non-uniform memory access) and MCDRAM (multichannel dynamic random access memory) configurations. These configurations are reported in the bios-populated hardware inventory table. At system boot time, xtbootsys invokes xthwinv to capture a snapshot of the XC system's hardware inventory, including KNL NUMA and MCDRAM modes, and publishes this snapshot for consumption by the SDB node. To keep the cached hardware inventory on the SDB node in sync with the actual state of the XC system, xtwarmswap --add can recapture hardware inventory for swapped blades and dynamically update the SDB node when new hardware is added. However, due to the dynamic reconfiguration capability of a KNL node, its apparent hardware complement (as reported by bios hardware inventory) can change on any node reboot.

This procedure describes how to use the capmc utility to reconfigure KNL nodes. When the reconfigured nodes are rebooted, the SDB node will be notified of those hardware inventory changes, thereby keeping the hardware inventory current. See the capmc(8) man page for details about these commands, including default and allowed values for setting the "mode" of a node.

  1. Change NUMA modes on one or more KNL nodes.
    1. List the NUMA parameters possible for a set of KNL nodes (specified as a comma-separated list and/or range of NIDs). 
      smw# capmc get_numa_capabilities --nids nidlist --pretty
    2. Change the NUMA configuration mode on a set of KNL nodes, as needed.
      smw# capmc set_numa_cfg --nids nidlist --mode mode
  2. Change MCDRAM modes on one or more KNL nodes.
    1. List the MCDRAM parameters possible for a set of KNL nodes (specified as a comma-separated list and/or range of NIDs). 
      smw# capmc get_mcdram_capabilities --nids nidlist --pretty
    2. Change the MCDRAM configuration mode on a set of KNL nodes (specified as a list of NIDs), as needed.
      For MCDRAM, the mode represents the amount of MCDRAM that is to be used as cache.
      smw# capmc set_mcdram_cfg --nids nidlist --mode mode
  3. Call capmc node_reinit to shut down the changed KNL nodes cleanly, go through a full BIOS re-initialization, and reboot those nodes.
    smw# capmc node_reinit --nids nidlist --reason "changed MCDRAM/NUMA on KNL"