The aprun Memory Affinity Options
How to optimize memory references.
A compute node's memory and CPUs are divided into one or more NUMA nodes. References from a CPU on one NUMA node to memory on another NUMA node can adversely affect performance. Cray has added aprun memory affinity options to give the user run time controls that may optimize memory references. Most nodes with an NVIDIA GPU have only a single NUMA node; Cray XK compute nodes with a GPU can have more than one NUMA node.
Applications can use one or all NUMA nodes of a Cray system compute node. If an application is placed using one NUMA node, other NUMA nodes are not used and the application processes are restricted to using NUMA-node memory on the node on which the application is placed. This memory usage policy is enforced by running the application processes within a cpuset. A cpuset consists of cores and local memory on a compute node.
When an application is placed using all NUMA nodes, the cpuset includes all node memory and all CPUs. In this case, each application process allocates memory from the NUMA node on which it is running. If insufficient free NUMA-node memory is available on the process node, the allocation may be satisfied by using NUMA-node memory from another node. In other words, if there is not enough NUMA node n memory, the allocation may be satisfied by using NUMA node n+1 memory. An exception is the -ss (strict memory containment) option. For this option, memory allocations are restricted to local-NUMA-node memory even if all remote-NUMA-node memory is available to the application. Therefore, -ss requests fail if there is insufficient memory on the local NUMA node regardless of how much memory is available elsewhere on the compute node.
The aprun memory affinity options are:
-S pes_per_numa_node-ss
Use these aprun options for each element of an MPMD application and vary them with each MPMD element as required. For complete details, see Run Applications Using the aprun Command.
Use cnselect numcores.eq.number_of_cores to get a list the Cray system compute nodes.
Use the aprun -L or qsub -lmppnodes options to specify those lists or a subset of those lists. For additional information, see the aprun(1), cnselect(1), and qsub(1) man pages.