Configure MAMU Nodes on the Cray XC System

Configure multiple application, multiple user (MAMU) nodes by repurposing compute nodes and configuring them.

Obtain the cnames of the compute nodes to be repurposed.

This procedure describes how to configure multiple application, multiple user (MAMU) nodes on the Cray XC system. It comprises the following tasks:
  • Create and modify a config set for MAMU nodes.
  • Create and update an image recipe for MAMU nodes.
  • Build and export the MAMU image.
  • Mark MAMU nodes as 'service' and map MAMU image and config set


–––––– CREATE AND MODIFY A CONFIG SET FOR MAMU NODES ––––––

  1. Create a CLE config set for use on repurposed compute nodes.
    Clone the CLE config set used on the PBS server host (SDB node), p0 in the example, to a config set for use on MAMU nodes, p0-mamu in the example.
    smw# cfgset create --clone p0 p0-mamu
  2. Create a node group in the cray_node_groups config service to contain the repurposed nodes.
    1. Create the new node group (mamu_nodes in the example) and verify that it was added to the list of node groups.
      smw# cfgset modify --add mamu_nodes cray_node_groups.settings.groups.data p0-mamu

smw# cfgset get cray_node_groups.settings.groups.data p0-mamu
    2. Set the description field of the MAMU node group and verify that it was set.
      smw# cfgset modify --set 'Repurposed Compute Nodes' \
cray_node_groups.settings.groups.data.mamu_nodes.description p0-mamu

smw# cfgset get cray_node_groups.settings.groups.data.mamu_nodes.description p0-mamu
Repurposed Compute Nodes
    3. Add the repurposed compute nodes to the MAMU node group and verify that they were added.
      This example adds two nodes to the MAMU node group. Adjust the number of nodes and their cnames for this system.
      smw# cfgset modify --add c0-0c0s7n2 --add c0-0c0s7n3 \
cray_node_groups.settings.groups.data.mamu_nodes.members p0-mamu

smw# cfgset get cray_node_groups.settings.groups.data.mamu_nodes.members
c0-0c0s7n2
c0-0c0s7n3
  3. Ensure that secure shell (SSH) is enabled for the system.
    smw# cfgset modify --set true cray_ssh.enabled p0-mamu

smw# cfgset get cray_ssh.enabled p0-mamu
true
  4. Add the MAMU nodes node group (mamu_nodes) to the node_groups_as_client list in the cray_rsip service, and then verify that it was added.
    smw# cfgset modify --add mamu_nodes cray_rsip.settings.service.data.node_groups_as_client p0-mamu

smw# cfgset get cray_rsip.settings.service.data.node_groups_as_client p0-mamu
  5. Add the MAMU nodes node group (mamu_nodes) to the config_id_service_groups list in the cray_auth service, and then verify that it was added.
    smw# cfgset modify --add mamu_nodes cray_auth.settings.access.data.config_id_service_groups p0-mamu

smw# cfgset get cray_auth.settings.access.data.config_id_service_groups p0-mamu
  6. Remove all default modules on the service nodes and verify that they were removed.
    smw# cfgset get cray_user_settings.settings.default_modules.data.service p0-mamu
sysadm
nodehealth
nodestat
sdb
alps
llm
system-config

    smw# cfgset modify --clear cray_user_settings.settings.default_modules.data.service p0-mamu

smw# cfgset get cray_user_settings.settings.default_modules.data.service p0-mamu
smw#
  7. Enable out of memory (OOM) App kill by configuring it in the system setting of the cray_sysenv config service.
    Create a new system entry called "vm_oom_kill" and then set and verify each of its fields.
    smw# cfgset modify --add vm_oom_kill cray_sysenv.settings.system.data p0-mamu

smw# cfgset get cray_sysenv.settings.system.data p0-mamu

    smw# cfgset modify --set 'Enable out-of-memory (OOM) app kill' \
cray_sysenv.settings.system.data.vm_oom_kill.description p0-mamu

smw# cfgset get cray_sysenv.settings.system.data.vm_oom_kill.description p0-mamu

    smw# cfgset modify --set sysctl cray_sysenv.settings.system.data.vm_oom_kill.type p0-mamu

smw# cfgset get cray_sysenv.settings.system.data.vm_oom_kill.type p0-mamu

    smw# cfgset modify --add mamu_nodes \
cray_sysenv.settings.system.data.vm_oom_kill.target_groups p0-mamu

smw# cfgset get cray_sysenv.settings.system.data.vm_oom_kill.target_groups p0-mamu

    smw# cfgset modify --set vm.oom_appkill cray_sysenv.settings.system.data.vm_oom_kill.name p0-mamu

smw# cfgset get cray_sysenv.settings.system.data.vm_oom_kill.name p0-mamu

    smw# cfgset modify --set 1 cray_sysenv.settings.system.data.vm_oom_kill.value p0-mamu

smw# cfgset get cray_sysenv.settings.system.data.vm_oom_kill.value p0-mamu
  8. Add a DVS client mount for the MAMU nodes.
    Use /home as the mount point. Create a new client mount entry called /home and then set and verify each of its fields.
    smw# cfgset modify --add /home cray_dvs.settings.client_mount.data p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data p0-mamu

    smw# cfgset modify --set /home cray_dvs.settings.client_mount.data./home.mount_point p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.mount_point p0-mamu

    smw# cfgset modify --set /home cray_dvs.settings.client_mount.data./home.spath p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.spath p0-mamu

    smw# cfgset modify --add tier2_nodes cray_dvs.settings.client_mount.data./home.server_groups p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.server_groups p0-mamu

    smw# cfgset modify --add mamu_nodes cray_dvs.settings.client_mount.data./home.client_groups p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.client_groups p0-mamu

    smw# cfgset modify --set false cray_dvs.settings.client_mount.data./home.loadbalance p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.loadbalance p0-mamu

    smw# cfgset modify --set 0 cray_dvs.settings.client_mount.data./home.attrcache_timeout p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.attrcache_timeout p0-mamu

    smw# cfgset modify --set true cray_dvs.settings.client_mount.data./home.readonly p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.readonly p0-mamu

    smw# cfgset modify --set 'maxnodes=1' cray_dvs.settings.client_mount.data./home.options p0-mamu

smw# cfgset get cray_dvs.settings.client_mount.data./home.readonly p0-mamu
  9. Create a host for each MAMU node in the config set for the MAMU nodes (p0-mamu in the example) and in the config set for the PBS server host (p0 in the example).
    In this example, there are two MAMU nodes, so two hosts are specified: ppn1 and ppn2. This example specifies two hosts, one for each node in the MAMU node group. Adjust the number of hosts and their host IDs for this system.
    1. Create a host for the first MAMU node in both config sets.
      smw# cfgset modify --add ppn1 cray_net.settings.hosts.data p0-mamu

smw# cfgset modify --add ppn1 cray_net.settings.hosts.data p0

      smw# cfgset modify --set 'Post-processing node' \
cray_net.settings.hosts.data.ppn1.description p0-mamu

smw# cfgset modify --set 'Post-processing node' \
cray_net.settings.hosts.data.ppn1.description p0

      smw# cfgset modify --set c0-0c0s7n2 cray_net.settings.hosts.data.ppn1.hostid p0-mamu

smw# cfgset modify --set c0-0c0s7n2 cray_net.settings.hosts.data.ppn1.hostid p0

      smw# cfgset modify --set ppn1 cray_net.settings.hosts.data.ppn1.hostname p0-mamu

smw# cfgset modify --set ppn1 cray_net.settings.hosts.data.ppn1.hostname p0

      smw# cfgset modify --set false cray_net.settings.hosts.data.ppn1.standby_node p0-mamu

smw# cfgset modify --set false cray_net.settings.hosts.data.ppn1.standby_node p0
    2. Create a host for the second MAMU node in both config sets.
      smw# cfgset modify --add ppn2 cray_net.settings.hosts.data p0-mamu

smw# cfgset modify --add ppn2 cray_net.settings.hosts.data p0

      smw# cfgset modify --set 'Post-processing node' \
cray_net.settings.hosts.data.ppn2.description p0-mamu

smw# cfgset modify --set 'Post-processing node' cray_net.settings.hosts.data.ppn2.description p0

      smw# cfgset modify --set c0-0c0s7n3 cray_net.settings.hosts.data.ppn2.hostid p0-mamu

smw# cfgset modify --set c0-0c0s7n3 cray_net.settings.hosts.data.ppn2.hostid p0

      smw# cfgset modify --set ppn2 cray_net.settings.hosts.data.ppn2.hostname p0-mamu

smw# cfgset modify --set ppn2 cray_net.settings.hosts.data.ppn2.hostname p0

      smw# cfgset modify --set false cray_net.settings.hosts.data.ppn2.standby_node p0-mamu

smw# cfgset modify --set false cray_net.settings.hosts.data.ppn2.standby_node p0
  10. Disable ALPS on the MAMU nodes by adding the MAMU node group to the mamu_node_groups setting of the cray_aps config service. 
    smw# cfgset modify --add mamu_nodes cray_alps.settings.common.data.mamu_node_groups p0-mamu


–––––– CREATE AND UPDATE AN IMAGE RECIPE FOR MAMU NODES ––––––

  1. Create a recipe for the MAMU nodes and add an existing recipe as a subrecipe.
    Create a MAMU recipe based on either the service node recipe or the login netroot recipes.
    • Service recipe: 
      smw# recipe create --description "MAMU service recipe" \
mamu-service_cle_7.0.up01_sles_15_ari

smw# recipe update -i service_cle_7.0.up01_sles_15_ari \
mamu-service_cle_7.0.up01_sles_15_ari
    • Login netroot recipes:

      Create two recipes for netroot, because two images are required: the initrd-mamu-large image will be set as the NIMS boot image, and the mamu-large image will be set as the NIMS "netroot" kernel parameter.

      smw# recipe create --description "MAMU login initrd netroot recipe" \
initrd-mamu-large_cle_7.0.up01_sles_15_ari

recipe update -i initrd-login-large_cle_7.0.up01_sles_15_ari \
initrd-mamu-large_cle_7.0.up01_sles_15_ari

      smw# recipe create --description "MAMU login netroot recipe" \
mamu-large_cle_7.0.up01_sles_15_ari

smw# recipe update -i login-large_cle_7.0.up01_sles_15_ari \
mamu-large_cle_7.0.up01_sles_15_ari
  2. Add the repository.
    smw# recipe update -r common_cle_7.0.up01_sles_15_ari mamu-service_cle_7.0.up01_sles_15_ari

smw# recipe update -r common_cle_7.0.up01_sles_15_ari mamu-large_cle_7.0.up01_sles_15_ari

smw# recipe update -p cray-xpmem -p cray-libxpmem0 -p cray-libxpmem-devel / 
-p cray-xpmem-kmp-cray_ari_s -p cray-libxpmem-devel-headers mamu-service_cle_7.0.up01_sles_15_ari

smw# recipe update -p cray-xpmem -p cray-libxpmem0 -p cray-libxpmem-devel /
-p cray-xpmem-kmp-cray_ari_s -p cray-libxpmem-devel-headers mamu-large_cle_7.0.up01_sles_15_ari
  3. Update the new MAMU node recipe by adding packages for MAMU. The -p option specifies each required package.
    • Service-based MAMU recipe:
      smw# recipe update -p cray-xpmem -p cray-libxpmem0 -p cray-libxpmem-devel \
-p cray-xpmem-kmp-cray_ari_s -p cray-libxpmem-devel-headers \
mamu-service_cle_7.0.up01_sles_15_ari
    • Login netroot-based MAMU recipe:

      Add packages to the mamu-large recipe only, not to the initrd-mamu-large recipe.

      smw# recipe update -p cray-xpmem -p cray-libxpmem0 -p cray-libxpmem-devel \
-p cray-xpmem-kmp-cray_ari_s -p cray-libxpmem-devel-headers \
mamu-large_cle_7.0.up01_sles_15_ari


–––––– BUILD AND EXPORT THE MAMU IMAGE ––––––

  1. Build an image using the updated MAMU recipe.
    • Service-based MAMU recipe:
      smw# image create -r mamu-service_cle_7.0.up01_sles_15_ari \
mamu-service_cle_7.0.up01_sles_15
    • Login netroot-based MAMU recipes:
      smw# image create --arch x86_64 -r initrd-mamu-large_cle_7.0.up01_sles_15_ari \
initrd-mamu-large_cle_7.0.up01_sles_15_x86-64

smw# image create --arch x86_64 -r mamu-large_cle_7.0.up01_sles_15_x86-64_ari \
mamu-large_cle_7.0.up01_sles_15_x86-64
      Note to reviewers: are the above image names correct? Some have _ari appended, others do not.
  2. Export the MAMU image.
    • Service-based MAMU image:
      smw# image export mamu-service_cle_7.0.up01_sles_15
    • Login netroot-based MAMU images:

      Export the initrd-mamu-large image, and push the mamu-large image root.

      Note that the following example uses image sqpush. If this image root will be modified and pushed more than once to test it, consider using image push instead. For more information, see About Image Pushes: push versus sqpush.

      smw# image export initrd-mamu-large_cle_7.0.up01_sles_15_x86-64

smw# image sqpush -d boot mamu-large_cle_7.0.up01_sles_15_x86-64


–––––– MARK MAMU NODES, MAP MAMU IMAGE AND CONFIG SETS ––––––

The examples in the remaining steps use the same two nodes from the MAMU node group created at the beginning of this procedure. Adjust the number of nodes and their cnames for this system.

  1. Specify for the Hardware Supervisory System (HSS) that the repurposed compute nodes are now MAMU service nodes.
    1. Halt each MAMU node.
      Run this command as crayadm.
      crayadm@smw> xtcli halt c0-0c0s7n2,c0-0c0s7n3
    2. Bounce each MAMU node.
      crayadm@smw> xtbounce c0-0c0s7n2,c0-0c0s7n3
    3. Display the Node Image Mapping Service (NIMS) metadata for each MAMU node to verify current node settings.
      crayadm@smw> cnode list c0-0c0s7n2 c0-0c0s7n3
      The listing should indicate that the current settings are compute type, compute node group, compute node image, the current active config set (p0), and various kernel parameters.
    4. Change the type of each MAMU node to service.
      crayadm@smw> xtcli mark_node service c0-0c0s7n2,c0-0c0s7n3
      Important. Using this command to change the node type in the HSS database is considered a hardware change, and to keep everything in synch (HSS, NIMS, and IMPS), the MAMU nodes must also have their NIMS parameters changed and the CLE and global config sets updated. These occur in later steps in this procedure.
  2. Create a NIMS map for use with the MAMU configuration and images.
    It is good practice to use a different NIMS map for each image-configuration combination. Clone the active NIMS map, then set the new MAMU map as the active map so that subsequent cnode commands will make changes to the MAMU map only.
    1. Find the NIMS map that is active.
      crayadm@smw> cmap list
    2. Clone the active NIMS map.
      crayadm@smw> cmap create --clone active_map mamu_map
      Trouble? The cmap command will first verify that the CLE config set associated with the NIMS map exists. If it does not exist, the command will fail with an error message to that effect.
      • If the config set is expected to be missing (for example, during an installation when the CLE config set has not yet been created), then repeat the cmap create command with the --no-verify flag.
      • If the config set is NOT expected to be missing, then create/locate the missing config set, set it as the default config set for the NIMS map, and repeat the cmap create command.
    3. Set the new MAMU map as the active NIMS map.
      crayadm@smw> cmap setactive mamu_map
  3. Update the NIMS parameters for the MAMU nodes.
    Remove the netroot_compute group and add the service group, set the MAMU image and config set, and remove or set the netroot kernel parameter, depending on whether the MAMU nodes have a netroot image.
    • Service-based MAMU nodes:

      The netroot kernel parameter must be removed in this case.

      crayadm@smw> cnode update -G netroot_compute -g service \
-i /var/opt/cray/imps/boot_images/mamu-service_cle_7.0.up01_sles_15 \
-c p0-mamu -K netroot c0-0c0s7n2 c0-0c0s7n3
    • Login netroot-based MAMU nodes:

      The netroot kernel parameter must be set to the mamu-large image root in this case.

      crayadm@smw> cnode update -G netroot_compute -g service \
-i /var/opt/cray/imps/boot_images/initrd-mamu-large_cle_7.0.up01_sles_15_x86-64 \
-c p0-mamu -s netroot=mamu-large_cle_7.0.up01_sles_15_x86-64 c0-0c0s7n2 c0-0c0s7n3
  4. Verify the NIMS parameters for the MAMU nodes.
    crayadm@smw> cnode list c0-0c0s7n2 c0-0c0s7n3
 NAME        TYPE     GROUP    IMAGE  CONFIG_SET  EXT_PARAMETERS
c0-0c0s7n2  service  service  /var/opt/cray/imps/boot_images/mamu-service_cle_7.0.up01_sles_15.cpio  p0-mamu
  sdbnodeip=10.131.255.253 bootnodeip=10.131.255.254 hsn_ipv4_mask=255.252.0.0 
  hsn_ipv4_net=10.128.0.0 NIMS_GROUP=service ids=10.128.0.6 config_set=p0-mamu
c0-0c0s7n3  service  service  /var/opt/cray/imps/boot_images/mamu-service_cle_7.0.up01_sles_15.cpio  p0-mamu
  sdbnodeip=10.131.255.253 bootnodeip=10.131.255.254 hsn_ipv4_mask=255.252.0.0 
  hsn_ipv4_net=10.128.0.0 NIMS_GROUP=service ids=10.128.0.6 config_set=p0-mamu


–––––– UPDATE/VALIDATE CLE CONFIG SETS AND REBOOT MAMU NODES ––––––

  1. Update both config sets that were modified.
    Because the cfgset modify command does not call pre- and post-configuration scripts, any config set modified in that way is marked as invalid. Therefore both of the config sets that were modified earlier in the procedure must be updated using cfgset update. The "prepare" mode is used so that no user interaction will be required.
    smw# cfgset update -m prepare p0-mamu
smw# cfgset update -m prepare p0
    This update of the CLE config set was also necessitated by the MAMU nodes being repurposed. The update ensures that the files/node_groups/platforms/p0-mamu.platform.json file in the global config set directory structure (/var/opt/cray/imps/config/sets/global) is updated with accurate information.
  2. Validate both config sets.
    smw# cfgset validate p0-mamu
smw# cfgset validate p0
  3. Reboot the MAMU nodes.
    This command must be run as crayadm.
    crayadm@smw> xtbootsys --reboot c0-0c0s7n2,c0-0c0s7n3