A modern high-performance interconnect for HPC and AI clusters

HPE Slingshot delivers industry leading performance, bandwidth, and low latency for HPC, AI/ML, and data analytics applications. It is Cray’s 8th generation interconnect architecture with unique and powerful innovations engineered from the silicon up to combine the best of a high-performance supercomputing-class interconnect with Ethernet interoperability. This enables a unified high-performing cluster that delivers great performance on both traditional modeling and simulation codes alongside native sockets-based applications, and with direct Ethernet connectivity to storage without requiring gateway nodes.

Beyond raw speed (25.6 Tbps of bidirectional bandwidth per switch and 200 Gbps endpoints), HPE Slingshot dramatically controls loaded latency, the true determinant of realized performance on tightly coupled HPC and AI workloads. Innovative congestion management reduces tail latency and run-to-run execution time variability, and fine-grained adaptive routing ensures high utilization of the bandwidth available.

With HPE Slingshot, the same interconnect in the world’s most powerful supercomputers can now be deployed across a wide spectrum of HPC and AI/ML clusters based on HPE Apollo, HPE ProLiant, and HPE Cray XD server systems. This document focuses on standalone HPE Slingshot components, systems, and software for such clusters. For information about HPE Slingshot blades and elements used in HPE Cray Supercomputing EX systems, please refer to the Cray EX QuickSpecs. In addition, please refer to the Supplemental Configuration and Integration Guide for in-depth guidance on designing and implementing HPC and AI/ML clusters with HPE Slingshot.

HPE Slingshot Highlights

High Bandwidth and Low Latency

HPE Slingshot switches provide 25.6 Tbps of bi-directional bandwidth with 64 ports each capable of 200 Gbps, delivering high bandwidth and low latency simultaneously, overcoming traditional Ethernet approaches to buffering and error handling that result in queueing latency and hinder low-latency RDMA applications.

Automatic Congestion Management

The HPE Slingshot network uniquely addresses the congestion challenges of short-lived, small packet flows typical in HPC applications. In hardware, each switch detects congestion, identifies its causes, and provides real-time feedback to its peers. The system distinguishes perpetrators from victims of injection and limits the injection rate from the perpetrator until congestion clears.

Fine-Grained Adaptive Routing

HPE Slingshot tracks real-time information on load across each switch-to-switch path, and dynamically re-routes traffic to balance load (rather than sticking only to a predetermined path) to improve network performance and bandwidth utilization. Uniquely, HPE Slingshot’s adaptive routing decisions are made on a packet-by-packet basis for unordered traffic, or a flow-by-flow basis for protocols (like TCP/IP) requiring ordered delivery.

Quality of Service

HPE Slingshot supports highly configurable quality-of-service (QoS) that provides the ability to control how network bandwidth is allocated to different classes of traffic and applications.

Low-Diameter Network

The high-radix 64-port HPE Slingshot switches enable low-diameter topologies to reduce the quantity of network equipment, cabling, power and cooling costs. The initial topology offered is a 3-hop dragonfly topology based on a mesh of all-to-all that further reduces latency and improves sustained bandwidth and reliability. Fine-grained adaptive routing and congestion management virtually eliminate any locality concerns related to job placement and storage-connectivity in the topology.

Ethernet Standards

The HPE Slingshot system interconnect implements a full range of Ethernet standards, with hardware support focusing on the data movement protocols, IPv4 and IPv6 in particular.

HPE Slingshot for HPC Clusters consists of the following elements:

HPE Slingshot Top-of-Rack Switch

Based on HPE-designed silicon, the high-performance HPE Slingshot switch provides 64-ports at 200 Gbps each for 25.6 Tbps of bidirectional bandwidth. Ports are configured either as edge ports for connectivity to supported Ethernet NICs or external routers at 100GE or 200GE, or as 200 Gbps fabric ports that link the switches together with enhanced resiliency and performance.

HPE Slingshot Network Interface Card

The HPE Slingshot Network Interface Card provides a single port of 200 Gbps connectivity to PCI-Express Gen 4-based host systems. It is based on HPE-designed silicon and is qualified in select HPE Apollo, HPE ProLiant, and HPE Cray XD servers. The HPE Slingshot NIC features high performance RDMA and extensive hardware acceleration for MPI and SHMEM based software. These HPC optimized features are exposed to software applications using the Libfabrics API interface for RDMA software. Additionally, the NIC supports a broad range of host software natively using the Linux “sockets” interface for IP software.

HPE Slingshot also supports select 100 Gbps and 200 Gbps standard-based Ethernet adapters as an alternative option.

Cables

HPE Slingshot cables are designed and validated to reliably support the demanding signaling speed used in 200 Gbps links. Copper cables support shorter lengths, and Active Optical Cables provide connectivity up to 100 m.

HPE Slingshot Fabric Management Suite software

The fabric manager provides configuration, monitoring, control, and networking functions for the entire HPE Slingshot network. This software runs on a standard ProLiant server (or a redundant pair of servers) and connects to the switches using the cluster’s out-of-band management network.

HPE Slingshot Top-of-Rack Switch

The HPE Slingshot Switch is streamlined from the silicon-level up for the highest throughput to minimize queueing latency, which is the main contributor to latency in practice. Optimizations ensure performance and reliability for the small granularity messages common in latency-sensitive HPC and AI workloads. For example, optimizations on links between switches reduce packet overheads and eliminate end-to-end timeouts and retransmissions that otherwise hinder use of Ethernet for latency sensitive applications. Meanwhile, adaptive routing and congestion control between switches ensure bandwidth is used extremely efficiently.

Each HPE Slingshot top of rack switch (TOR) contains 1 HPE Slingshot switch ASIC and supplies sixty-four 200 Gbps ports via 32 QSFP double density (QSFP-DD) connectors in a 1U 19” standard rack-mount form factor chassis. Each QSFP-DD physical port can provide either 2 x 100 Gbps or 2 x 200 Gbps network links. Ports can be configured to connect to HPE Slingshot switches or to edge devices including select HPE or 3^rd party enterprise Ethernet switches or supported NICs. Redundant N+1 hot swappable power supplies and fans ensure availability. Management and monitoring operations utilize the cluster’s out-of-band management network.

HPE Slingshot 200 Gbps NIC

The HPE Slingshot SA210S Ethernet 200 GB 1-port PCIe NIC works with the HPE Slingshot Top-of-Rack Switches in supported HPE Apollo, ProLiant, and Cray XD rack servers. It delivers supercomputing-class performance on challenging latency-sensitive HPC codes without compromise while also running standard IP applications using native Ethernet IP protocols. The card supports either QSFP-56 200 Gbps (4×50 Gbps PAM 4) and QSFP-28 100 Gbps (4 x 25 Gbps NRZ) Ethernet standard connectivity.

Many features accelerate HPC and AI performance. MPI message matching offload overlaps communications and compute and frees up host memory bandwidth and CPU resources. The NIC reduces compute overhead for GPU initiated communications by allowing the host CPU to set up operations that the GPU can trigger asynchronously. The efficient command launch mechanism reduces latency by receiving and pipelining commands without reading host memory. And a reliable delivery mechanism detects packet loss and enables end-to-end retry in hardware to reduce software overhead.

Some HPE Slingshot fabric capabilities are extended to the NIC-to-switch port link when using the HPE Slingshot NIC versus industry standard NICs including enhanced link resiliency and fine-grained flow control. The latter helps the automatic congestion apply stiff backpressure to the specific flow causing congestion which delivers granularity over backpressure to the entire port.

The HPE Slingshot NIC exposes RDMA and HPC optimized features to software using the Libfabric software interface.

Ordering Information:

Consult HPE Slingshot for HPC Clusters reference architecture information for supported host systems.

HPE Slingshot Fabric Management Suite

The HPE Slingshot Fabric Management Suite provides the management, control, and network functions and is required to configure and operate the HPE Slingshot network. The software is installed on a standard server that communicates with the switches via a 1 or 10 Gbps management network. This separation of data and control planes allows for independent reliable data transfers and management operations. All network management traffic is encrypted with switch-based certificates.

The Fabric Management Suite supports comprehensive health checking, monitoring and logging, performance profiling, and diagnostics that assure high performance with high availability and effective operational processes. Event, error, and counter data is gathered from all the switches in the network. Open documented APIs and a scriptable command line interface allow for integration with cluster management and network monitoring tools. All of these APIs support AuthN based on client certificates.

Scaling, efficiency, and performance of management and operations are achieved via a hierarchical distributed control plane design. Multiple Fabric Management Suite nodes can optionally operate in an Active/Standby mode to deliver a high-availability configuration.

Ordering Information:

Consult HPE Slingshot for HPC Clusters reference architecture information for current details on supported host systems and recommended configurations.

HPE Slingshot Cables

HPE validates HPE Slingshot cables to ensure reliability and performance running up to 200 Gbps with the switch and supported adapters. Because HPE Slingshot switches use double density ports, each QSFP-DD cable-end carries two 200 Gbps links, often reducing the total cable count to deploy. HPE offers both copper and Active Optical (AOC) cables with various lengths. However, certain restriction may apply for using copper cables for switch-to-switch or switch-to-adapter connections. Consult reference architectures for specific wiring guidelines that must be followed for proper operation.