Cray XC System Configuration

Overview of how Cray XC systems are configured using config sets, the configurator, IDS, Ansible, etc.

To configure Cray XC systems and manage configuration content, system administrators use the Cray configuration management framework (CMF). The CMF comprises configuration data, the tools to manage and distribute that data, and software to apply the configuration data to the running image at boot time. Its major components include configuration service packages, config sets, the IMPS distribution service (IDS), the configurator, cray-ansible, and Ansible.

Configuration Starts with Configuration Service Packages

Configuration content (data and software) is installed as configuration service packages on the management node of Cray XC systems (in /opt/cray/imps_config/<service package>/default/configurator by default). Each service package delivers configuration content for one or more system services. The contents of each service package reside in the following subdirectories:

ansible: Drop zone for Cray-provided Ansible play content.
callbacks: Pre- and post-configuration scripts.
dist: Drop zone for other Cray-provided content, such as static files required for the configuration of a service.
template: Configuration templates that define the configuration settings to be set and provide some default values. These templates are never modified by administrators or other users.

Configuration service packages are installed for system upgrades and updates as well as for initial installation.

Configuration Information is Stored in Config Sets

Administrators use the cfgset command to manage configuration information. It takes configuration content delivered in service packages and invokes the configurator tool to combine that content with site-specific configuration content gathered from administrators either interactively or through bulk import. The results are used by cfgset to create a configuration set or config set. A config set is a central repository that stores all configuration information necessary to operate the system. Config sets reside on the management node (e.g., the SMW) in /var/opt/cray/imps/config/sets by default. The contents of each config set reside in the following subdirectories:

ansible: Drop zone for local site-provided Ansible play content to be distributed with the config set. When the config set is created, cfgset copies Ansible content from service packages to this location. Whenever the config set is updated, cfgset copies Ansible content from service packages again, overwriting the previous service-package Ansible content and leaving the site-provided content unchanged.
changelog: YAML change logs from previous sessions with the configurator.
config: Configuration templates containing configuration information. When the config set is created, the configurator copies service package templates to this location. Administrators can modify the content of these templates using cfgset and the configurator. Whenever the config set is updated, the configurator merges service package templates with the templates in this location.
dist: Drop zone for other site-provided content, such as static files required for the configuration of a service. When the config set is created, cfgset copies dist content from service packages to this location. Whenever the config set is updated, cfgset copies dist content from service packages again, overwriting the previous service-package dist content and leaving the site-provided content unchanged.
files: Files necessary for system configuration that are generated by configuration callback scripts or manually and distributed with the config set (e.g., /etc/hosts).
worksheets: Configuration worksheets generated by the configurator using data stored in the configuration templates in the config subdirectory of the config set. Administrators copy these worksheets to a location outside the config set, edit them with site-specific configuration data, and then import them to create a new config set or update an existing one.

An administrator may create multiple config sets to support partitions or alternate configurations. Typically a config set of type cle is created for each partition to store partition- and CLE-specific content, and another config set of type global is created to store management node and global configuration data.

IDS Distributes Config Sets to Nodes

IDS, a read-only network share of content from the management node to the rest of the system, distributes config sets to every node in the system. All config sets are shared throughout the system, but only one cle config set is active on a given node at a time (in addition to an active global config set, which is applied to the entire system). Currently, IDS leverages the 9P network file system and the Linux automounter facility as its distribution mechanism; however, the content and use of the config sets is independent of the distribution mechanism.

Ansible Plays Apply Configuration during System Boot

Prior to booting the system, each node will have an image, the global config set, and the cle config set. When the system boots, each node boots an unconfigured software image. Then Ansible plays, which can be located in both the image and the config set (config set is the preferred location for site-supplied Ansible plays), apply configuration to that image, bringing up the services pertinent to each node.

Administrators Configure/Reconfigure the System on an Ongoing Basis

Configuration happens at times other than initial installation. New configuration service packages can be installed during system upgrades and updates, sites can decide to enable a new service or change the configuration of an existing service, and so forth. In all of these scenarios, an administrator uses the cfgset command to manage config sets and the cray-ansible script to apply any configuration changes. The cfgset command and its associated subcommands and options enable administrators to perform a variety of operations on config sets in addition to create and update, such as search, diff, list, show, validate, push, and remove. See the cfgset man page for a description of its subcommands and options and some examples of each.