HPE Cray Supercomputing EX 2500 System Details

A single rack can accommodate up to 24 compute blade slots within 3 compute chassis. In this configuration, the rack is not configured with any cooling fans. All cooling needs for the cabinet are provided by direct liquid cooling and the CDUs. This approach to cooling provides greater efficiency for the rack-level cooling, decreases power costs associated with cooling (no blowers) and utilizes a single water source per CDU

One rack supports the following:

• – Up to 3 compute chassis
• – Up to 3 power shelves with 4 rectifiers per shelf - 12 total 15 kW rectifiers per rack.

  • Notes: 1 rectifier per shelf is used for redundancy
• – 3 PDUs (1 per power shelf)
• – 3 power input whips (3-phase)
• – Maximum of 24 compute and/or accelerator blades
• – Maximum of 24 HPE Slingshot switch blades
• – Up to 2 in-rack CDUs for a max of 145KVA cooling capacity

Notes: 1 CDU is included by default in support of one compute chassis and will not show up as a line item in the configuration. The second CDU will show up as a line item in the configuration once two or more chassis are added to the configuration.

HPE Cray Supercomputing EX4252 Gen 2

The features of this compute blade are as follows:

• Four 2-socket CPU nodes
• AMD 4th Gen EPYC™ (Genoa - Zen4 up to 96 Cores)
• AMD 4th Gen EPYC™ (Bergamo - Zen4c up to 128 Cores)
• AMD 5th Gen EPYC™ (Turin Dense 400W – Zen5/5c up to 160 Cores)
• AMD 5th Gen EPYC™ (Turin Dense 500W – Zen5/5c up to 192Cores)
• 12 DIMMs per CPU socket (1DPC)
• Up to 64 GB DIMMs at up to 4800 MT/s for Genoa and Bergamo Processors
• Up to 64 GB DIMMs at up to 6400 MT/s for Turin 400W / 500W Processors
• Up to 8 HPE Slingshot 200Gbit/sec ports per blade
• 0 or 1 local NVMe M.2 SSD per node (up to 4 per blade)
• 1 Board Management Controllers (BMC) per blade Cooled with cold plate

HPE Cray Supercomputing EX254n

The features of this accelerator blade are as follows:

• – Two 4-socket Nvidia GH200 Grace Hopper Superchip nodes
• – 96GB HBM3 per GPU; 120GB LPDDR per CPU
• – Up to 8 HPE Slingshot 200Gbit/sec ports per blade
• – 0 or 1 local NVMe M.2 SSD per node (up to 2 per blade)
• – 2 Board Management Controllers (BMC) per blade
• – Cooled with cold plate

Switch Chassis

The Slingshot switch blades are in the switch chassis and mounted to the rear of the compute chassis. The purpose of the switch chassis is to provide a structure for orthogonally mounting the switch blades to the compute chassis. There is no active backplane connecting the switches to the compute and/or accelerator blades. Each compute blade directly connects to one or more switch blades in the switch chassis using a cableless connection. The switch chassis supports a maximum of eight Slingshot switch blades.

HPE Slingshot Switch Blade

The following switch blade types are supported

The HPE Slingshot Switch blade is a 64 x, 200Gb/s port switch designed for the switch chassis. Switch blades are inserted into the rear of the cabinet and provide the high-speed network interface for the compute and/or accelerator blades. Each switch blade connects to all eight compute and/or accelerator blades through orthogonal connectors and provides fabric connections using copper and optical cables through its faceplate to expand the network.

Each switch blade has 8 local connectors, with two ports per connector for a total of 16 downlink ports to the eight compute and/or accelerator blades. Each switch blade also has 24 QSFP-DD connectors with two 200 Gb/sec ports per connector for a total of 48 copper or optical connections to the other switch blades in the cabinet and other cabinets to create the high-speed fabric. The number of switch blades in a chassis depends on the number of NIC injection points per compute node. For two-socket CPU blades such as the HPE Cray EX425 or EX4252 or EX420, a typical configuration would be two switch blades in the switch chassis to support the 4 nodes on the compute blade via the two port NIC Mezzanine card. These two switches would service this configuration across all 8 compute blades in the chassis which consists of 32 nodes. For accelerator-based blades, a typical configuration is four switch blades in order to serve one NIC per GPU.

HPE Slingshot NIC Cards

Compute and/or accelerator blades are outfitted with a mezzanine card that provides compute blade NIC connectivity to the Slingshot switch blades. Each mezzanine card supports two NICs through two PCIe buses. Each mezzanine card is 100% direct liquid cooled through a cold plate. An “L0” copper cable connects each NIC mezzanine card to the connector on the back of the compute blade. The compute blade will be the determinant for which mezzanine cards are supported and the number of mezzanine cards that are required per blade and whether there are options to configure the injection bandwidth per node (meaning the number of NICs).

The following mezzanine card type are offered:

Dual port 100Gbit/sec Mellanox ConnectX-5 Ethernet Mezzanine Card

This mezzanine card is a two-port NIC design. Each of the two 100 Gb/s Ethernet NIC port connects to a host system using a PCIe Gen3 a x16 channel. Solutions with this NIC can utilize programming environment support for HPE Cray MPI, OpenSHMEM, Chapel, UPC, as well as third-party software that utilizes the RDMA-Over-Converged-Ethernet (RoCE) protocol.

Dual port 200 Gbit/sec HPE Slingshot Mezzanine Adapter

The HPE Slingshot 200 Gbit/sec mezzanine card is based on HPE silicon designed to deliver supercomputer-class performance and scalability with the HPE Slingshot Switches. The NICs incorporate hardware features that extensively offload message processing across the range of HPC workloads and message sizes to ensure compute resources operate with high efficiency due to less interruption and latency from message and communications processing. The NICs also seamlessly complement operation with high performance networking technologies built into the Slingshot switches to optimize the end-to-end solution for congestion management, fine grain flow control, and traffic classes. The host interface is PCIe Gen4 x16 with support for extended speed mode at 25GT/s (where supported by the CPU or GPU).

A full suite of HPC and Ethernet features is supported including strong progression of MPI message matching and data transfer, along with programming environment support for HPE Cray MPI, OpenSHMEM, Chapel, UPC, and industry software that can support the HPE Slingshot NIC through the Libfabric interface.

Topology

The HPE Cray Supercomputing EX Supercomputer operates using the dragonfly topology for the high-speed network fabric created using the Slingshot switches. This provides a lower cost, efficient, and highly scalable approach over alternative topology designs. Any two endpoints in even the largest supercomputer can be reached with 3 or fewer switch-to-switch hops which reduces latency. The design maximizes use of high-speed copper cables by utilizing all-to-all connected switches in an EX cabinet as a dragonfly “group”, each of which in turn uses optical cables to connect all the groups in different cabinets. This creates substantial efficiency by reducing the use of expensive optical cables and enabling multiple paths to deliver very high global bandwidth.

Various dragonfly switch group sizes are available and selected depending on the requirements to optimize across lowest system cost (by maximizing the number of copper cables), maximum system scale, and the desired scalable unit of deployment. For EX4000 configurations, it is typical to configure the entire cabinet as one switch group. Here the two typical configurations are the 16 switch group (2 switches in each of the 8 chassis in a cabinet) or a 32 switch group 4 switches in each of the 8 chassis in a cabinet). It is possible to implement more than one group in a cabinet, such as configuring a cabinet that had 4 switches in each of the 8 chassis in the cabinet as two 16-switch groups instead of a single 32-switch group.

In the EX2500 cabinet, because the maximum system size is usually smaller than for EX4000. Therefore, it is typical to configure each of the chassis as a group, although for larger systems build with EX2500, groups sizes that aggregate switches in several chassis together - within the same or adjacent racks – are supported.

The chart below provides a reference for the group sizes supported:

CDU (Cooling Distribution Unit) for EX4000 and EX2500

The cooling distribution unit (CDU) is a liquid-to-liquid heat exchanger that is used to remove heat from HPE Cray EX Supercomputer. The CDU uses a secondary loop to circulate a heat transfer liquid to the cold sinks. The heat captured in the secondary loop is transferred to the facilities primary loop via a liquid-to-liquid heat exchanger.

The CDU is designed to circulate and control the heat transfer fluid to the manifolds that are in each chassis in the cabinet. The in-row CDU is rated for 1.6 MW of cooling. One CDU supports a maximum of four cabinets. The 4U in-rack CDU is rated for 70kW of cooling and up to two (2) can be installed in an EX2500 rack providing up to 140kW of cooling.

The CDU consists of a cabinet (or a 4U chassis) that includes a heat exchanger, circulating pumps, control valve, sensors, controller, valves, and piping. The CDU monitors room conditions and prevents condensation by maintaining the secondary loop at a temperature above the room’s dew point.

All functions, such as switching pumps (if applicable), controlling water temperature, etc., are managed by the controller using user defined settings.

Software Stack

HPE Cray EX supercomputers are complete solutions with software and hardware that are tightly integrated and performance-tuned to offer the best system performance while bringing new standards in flexibility, manageability, and resiliency to supercomputing.

HPE Cray supercomputer software stacks address the needs of both system administrators, developers, and end users.

Administrative Software

HPE Cray Supercomputing EX supercomputer users now have the option to choose either HPE Cray System Management or HPE Performance Cluster Manager.

HPE Cray System Management - a built-for-scale system management solution offering administrators all functionalities they need to keep the HPE Cray EX system healthy, utilized to the maximum and accommodating wide range of workload requirements via –aaS experience. The software is built to manage systems which can scale to Exascale deployments featuring:

• – Comprehensive monitoring and management of all aspects of the system: CPU/GPU, network (integrated HPE Slingshot Fabric Manager), storage as well as power management and monitoring combined with provisioning for operational efficiency.
• – Partitioning and batch or container orchestration enable customers to run a variety of HPC/AI/HPDA workloads the way that makes the best use of their system without logistical constraints.
• – REST APIs & standard protocols enable full interoperability with existing monitoring, management, andautomation toolsets.

HPE Performance Cluster Manager - a comprehensive, flexible HPC system management solution that enables fast setup, provisioning and monitoring including the following features

• – Hardware discovery and Linux operating system installation for compute and service nodes
• – Inventory management
• – Telemetry data collection and analysis
• – Alert monitoring and component diagnosis
• – Power resource monitoring and management
• – Software image management Developer Software

HPE Cray Programming Environment – is a fully integrated software development suite offering programmers comprehensive set of tools for developing, porting, debugging, and tuning of their applications so they can shorten application development time and accelerate their performance.

The programming environment is designed to make porting of existing applications easier with minimal recording and changes to the existing programming models to simplify transition to the new hardware architectures and configurations, such as HPE Cray EX systems.

Operating System

HPE Cray OS is a compute operating system based on SLES with enhancements. The enhancements provide customers with capabilities specific to supercomputing and high-performance computing fully supported by HPE Services. These modifications don't alter the ability to run standard Linux applications, but rather enhance it for performance, scale, and reliability. We integrate and test these materials together and package releases.

Overall

While HPE Cray System Management and HPE Cray Operating System are designed to support HPE Cray EX systems with HPE Slingshot, HPE Cray Programming Environment product also supports other HPE and HPE Cray HPC systems (using InfiniBand interconnect).

The software stack is supported by HPE Services.

Features

Hardware Maintenance Service Features

The HPE Cray EX system benefits from HPE’s highest level of support for high-performance compute ‘HPE Complete Care - Cray’ that may include HPE presence onsite.

• – This service level offers customers access to the HPE Cray customer portal. Case logging is available 24x7 by telephone or via this customer portal.
• – There is a choice of two maintenance coverage windows: 9x5 or 24x7. Onsite response time options are Next Business Day, 4 hours, 2 hours, or 1 hour. When an issue is reported, an HPE technical representative will arrive onsite within the response time window to identify and begin resolving the issue.
• – HPE provides critical spare parts to reduce any downtime associated with failures or maintenance. Critical spare parts may be located either onsite or at professionally managed regional spare part depots that provide rapid transportation of spare parts to customer sites. Customers may elect to supplement the HPE-owned spare parts inventory by purchasing additional spare parts.
• – HPE Cray EX customers have access to the support snapshot analyzer that collects, analyzes, and reports support information for HPE air cooled and liquid cooled HPC, and HPE Cray ClusterStor systems.
• – HPE Remote Support provides remote access and support. Capabilities range from a customer having full control of a remote screen-sharing session to the HPE Services team having the ability to log in securely as needed to resolve issues and perform administrative functions.

Software Support Features

Support for HPE developed software includes the following features:

• – Access to self-help resources on customer portal

  • Ability to open and submit a support case
  • Access to HPE knowledge articles
  • Ability to download:
  • ο Software releases and updates, including BIOS and FW
  • ο Software Patches
• – Notification of key operational items through the field notice (FN) process
• – Assistance from HPE Services Services to resolve issues within the service level coverage window for the hardware contract; assistance includes:

  • Triage to investigate/analyze issues
  • Confirmation whether the issue is hardware or software

Confirmation if the issue is related to an HPE-supported product or a third-party-supported product. If the issue is with a HPE-supported product, HPE Services Servicesmay provide configuration recommendations, possible work arounds, and directions to install a later version or patch, and/or submit a bug to get the issue fixed. For HPE products, HPE reserves the right to determine whether and how an issue will be resolved.

Customized Software

Support is provided for products sold by HPE and with a valid HPE Services Services support agreement. Support for third-party products without a related HPE Services Services support agreement requires the user to contact the third-party vendor for assistance. If customers modify HPE-delivered software without authorization from HPE, any issues resulting from the unapproved modifications fall outside of the standard support service agreement and HPE is not responsible for any resulting defects, damage, failure, performance degradation, or issues of any kind, or correction or remedy of same. HPE may require the user to remove custom modifications to confirm that a modification is not the source of the issue. Customers may request that HPE Services Services assist in making modifications to a product. HPE Services Serviceswill do its best to implement the request via a billable statement of work (SOW).

API and CLI Support

Support is available for HPE published APIs. Unpublished APIs are not eligible for support. Documentation outlining published API best practices and limitations is available at support.hpe.com , accessible either directly or through the HPE Cray customer portal. HPE will assist in determining if the API is working correctly, if the documentation is incorrect, or if the issue is an enhancement request.

HPE Application Programming Interface (API) and Command Line Interface (CLI) features allow the flexibility to configure and customize your system to optimize operations in your environment. These tools have the ability to significantly alter your system operations. If not properly tested and implemented in a controlled manner, they can introduce significant problems in your environment. When using these features or otherwise modifying or altering APIs, customers take on the responsibility to resolve or mitigate any issues they have introduced into the system.

HPE Services Servicesis not available to provide support to resolve issues that arise from the use of CLIs or APIs in a form not identical to those published by HPE.

Customer Training

Training courses are taught by HPE system experts and combine lectures with hands-on labs to enhance understanding and retention. The courses cover all aspects of using and maintaining an HPE system, from system administration to application development, porting, and optimization. A full listing of the standard HPE Cray training courses, along with their descriptions, can be found at

https://education.hpe.com/ww/en/training/portfolio/servers.html.

Subject to separate ordering arrangement, classes are scheduled on regular cycles at the HPE training facilities and can be scheduled for onsite delivery. HPE also offers customized training courses and can provide quotes for these courses based on the customer’s needs.