Interconnects: Scale-Up and Scale-Out

In the 2018 Field Manual this chapter was about choosing between FDR and EDR InfiniBand for an MPI cluster. In 2026 the question is bigger. A single training job can span thousands of accelerators, and every step of that job ends in a collective operation that stalls every GPU until the slowest link finishes. The interconnect is no longer plumbing between computers; for distributed training and large-model inference it is part of the computer. This chapter walks the whole stack: the scale-up fabrics inside a node or rack (PCIe, NVLink, Infinity Fabric, UALink, TPU ICI), the scale-out fabrics between racks (InfiniBand, Ethernet with RoCE, Ultra Ethernet, Slingshot), the RDMA and collective software that rides on them, and the practical work of testing and designing a fabric for a research cluster. For the accelerators themselves see Accelerators; for how parallelism strategies map onto these fabrics see Training at Scale.

Why the Network Is the Computer

Classic HPC taught us that tightly coupled MPI codes live or die on network latency. Modern AI training adds a second pressure: enormous, synchronized bursts of bandwidth. Data parallel training exchanges gradients every step. Tensor parallelism exchanges activations inside every layer. Mixture-of-experts (MoE) models shuffle tokens between experts with all-to-all traffic. Each of these is a collective, and a collective finishes only when the last participant finishes. One slow link, one congested switch port or one misconfigured NIC sets the pace for the whole job.

Bandwidth-Bound vs Latency-Bound Collectives

A useful mental model is the alpha-beta model: the time to move a message of size S is roughly a fixed startup cost (alpha, the latency of software, NIC, switches and wire) plus S divided by bandwidth B. Small messages are dominated by alpha; large messages are dominated by S/B.

NVIDIA's nccl-tests documentation makes the bandwidth term concrete for all-reduce. With n ranks each having bandwidth B, a ring or tree all-reduce of S bytes needs 2(n-1) transfers per element, so the best achievable time is t = (S/B) x (2(n-1)/n). That is why nccl-tests reports a "bus bandwidth" (busbw = algbw x 2(n-1)/n) that you can compare directly to the hardware link rate. Reduce-scatter and all-gather each do half the work, with busbw = algbw x (n-1)/n.[1] Two consequences follow:

• Large gradient all-reduces are bandwidth-bound. Once messages reach tens or hundreds of megabytes, per-GPU injection bandwidth and fabric bisection bandwidth decide step time.
• Small messages are latency-bound. The alpha term is paid on every step of a ring, so it grows with the number of ranks. Tensor parallel activations, MoE token dispatch at small batch sizes, inference decode traffic, and classic MPI halo exchanges and MPI_Allreduce of a few doubles all live here.

Field note: When someone says "the network is fast enough," ask "for what message size?" A fabric that delivers line rate at 1 GiB can still be the bottleneck for an 8 KiB tensor parallel exchange or a latency-sensitive CFD code. Always sweep message sizes when you benchmark (the nccl-tests -b/-e/-f flags exist for exactly this).[2]

Scale-Up vs Scale-Out

Vendors now split the interconnect into two tiers, and the distinction drives most design decisions:

• Scale-up fabrics connect accelerators inside a node or rack with memory-semantic (load/store) access, very high bandwidth per GPU and low latency. Examples: NVLink and NVLink Switch, AMD Infinity Fabric and UALink, Google ICI.
• Scale-out fabrics connect nodes or racks with message-based RDMA over NICs and switches. Examples: InfiniBand, Ethernet with RoCEv2 or Ultra Ethernet transport, HPE Slingshot.

The per-GPU bandwidth gap between the tiers is large. A fifth-generation NVLink GPU has 1,800 GB/s of NVLink bandwidth,[8] while a GB300 NVL72 gives each GPU 800 Gb/s (100 GB/s) of scale-out network connectivity through ConnectX-8.[9] Parallelism strategies are arranged so the chattiest traffic (tensor and expert parallelism) stays inside the scale-up domain and the least chatty (data and pipeline parallelism) crosses the scale-out network.

Scale-Up Fabrics

PCIe Generations

PCIe is still the universal attach point: host CPU to GPU, GPU to NIC, NIC to NVMe. It sets the ceiling on how fast a NIC can feed a GPU, and in PCIe-only servers it is also the GPU-to-GPU path. PCI-SIG doubles the per-lane rate each generation.

Note: PCI-SIG publishes per-lane raw rates for Gen4 and Gen5 and states that a Gen5 x32 link provides 128 GB/s in each direction.[3] The Gen4 and Gen5 x16 figures above are that arithmetic (16 lanes x rate / 8 bits, per direction, raw). Gen6, Gen7 and Gen8 x16 figures are as PCI-SIG states them.

In practice: ConnectX-8 SuperNICs use a PCIe Gen6 x16 host interface (Gen5 compatible),[24] and NVIDIA lists 256 GB/s bidirectional PCIe Gen6 for both Blackwell and Rubin GPUs.[10] An 800 Gb/s NIC is 100 GB/s per direction, which a Gen5 x16 slot cannot carry at full rate. That is the reason 800G NICs and Gen6 platforms arrived together.

NVLink and NVSwitch

NVLink is NVIDIA's direct GPU-to-GPU link; NVLink Switch (NVSwitch) chips connect many links so every GPU can talk to every other GPU at full NVLink speed. NVIDIA's current specification table:[8]

Note: all figures from NVIDIA's NVLink page, which labels them preliminary and subject to change.[8] NVIDIA describes the per-GPU figure as bidirectional.[8][10]

Inside an eight-GPU HGX or DGX board, NVSwitch chips give an all-to-all NVLink domain. The DGX B200, for example, has two fifth-generation NVLink switches providing 14.4 TB/s of aggregate bandwidth.[12] NVIDIA says sixth-generation NVLink delivers over 14x the bandwidth of PCIe Gen6.[8]

Rack-Scale NVL72 Domains

The biggest architectural change since the last edition is the rack-scale NVLink domain. A GB300 NVL72 rack (listed by NVIDIA as available now) connects 72 Blackwell Ultra GPUs and 36 Grace CPUs through NVLink Switch trays, with 130 TB/s of NVLink bandwidth in the rack. Each GPU also gets 800 Gb/s of scale-out networking from ConnectX-8.[9] At launch NVIDIA announced the Vera Rubin NVL72 at 260 TB/s of NVLink bandwidth with 3.6 TB/s per GPU; the current product page lists 216 TB/s and 3 TB/s per GPU, so treat these preliminary figures with care[60].[8][10] NVIDIA stated in January 2026 that Rubin was in full production with partner products available in the second half of 2026.[11]

Why it matters: 72 GPUs that can load and store each other's HBM at NVLink speed means tensor and expert parallel groups can be far larger than eight before touching the slower scale-out network. The trade-off is that the NVL72 is a liquid-cooled, very high-density rack that dictates facility design (see Power, Cooling and Facilities) and is managed as one system, not 18 independent servers.

AMD Infinity Fabric, xGMI and UALink

AMD's GPU-to-GPU fabric is Infinity Fabric (the link layer is often called xGMI in software and tools; RCCL documentation describes RCCL as using "PCIe and xGMI high-speed interconnects").[44] On an eight-GPU MI355X platform each OAM module has 7 Infinity Fabric links at 153 GB/s peak each, plus a PCIe 5.0 x16 host interface.[13] Unlike an NVSwitch board there is no switch chip in this per-link bandwidth figure, so check how your baseboard wires the links: a GPU that talks to only one peer gets the bandwidth of the link(s) to that peer, not the aggregate. Collectives that spread traffic across all peers make the best use of it.

UALink is the open, multi-vendor answer to NVLink Switch. Status per the UALink Consortium:

• The UALink 200G 1.0 specification was released in April 2025. It defines 200G-per-lane scale-up connections for up to 1,024 accelerators in a pod, using a load/store protocol that the consortium describes as having "the same raw speed as Ethernet with the latency of PCIe switches."[16]
• In April 2026 the consortium ratified the Common Specification 2.0 (adding In-Network Compute), a Chiplet Specification 1.0 (UCIe 3.0 compliant) and a Manageability Specification 1.0. All specs are publicly downloadable.[17][18]

The first large deployment vehicle is AMD's Helios rack, launched in July 2026. It connects 72 MI455X GPUs using UALink over Ethernet (UALoE), with up to 260 TB/s of aggregate scale-up bandwidth; each GPU connects through 18 independent UALoE stations spread across switch trays for resilience.[14] AMD lists 43 TB/s of scale-out bandwidth per Helios rack,[15] using standards-based Ethernet aligned with the Ultra Ethernet Consortium for scale-out.[58]

Google ICI and Optical Circuit Switching

Google's TPUs take a different path. Every TPU has a built-in router and Inter-Chip Interconnect (ICI) links to its neighbors in a torus, with no external packet switch. Starting with TPU v4, Google groups chips into 4x4x4 cubes wired electrically and connects the cube faces through optical circuit switches (OCS) built on MEMS mirrors. The TPU v4 paper describes 48 OCSes (136x136 ports each) joining 64 cubes into a 4,096-chip machine, and reports that the OCS layer costs under 5% of system capital cost and under 3% of power. Because the OCS is effectively a programmable patch panel, it can route around failed hosts, carve isolated slices for different users, and build "twisted" tori that improve all-to-all bandwidth.[20]

For the current generation, Google's documentation lists TPU7x (Ironwood) at 9,216 chips per pod in a 3D torus, 1,200 GBps of bidirectional ICI bandwidth per chip (200 GBps per axis), and 100 Gbps of data center network (DCN) bandwidth per chip for traffic between pods.[19] Note that ratio: the scale-up (ICI) tier has far more bandwidth than the scale-out (DCN) tier, the same pattern as NVLink vs InfiniBand. See Cloud and Hybrid for how TPU slices are requested.

Scale-Out Fabrics

InfiniBand Generations

InfiniBand is a purpose-built, lossless, credit-flow-controlled fabric with a hardware transport and a central subnet manager. It remains the most common interconnect on the TOP500: the IBTA reports InfiniBand in 293 systems (59%) on the June 2026 list, with Ethernet RoCE in another 83.[22] A standard "4x" InfiniBand port bundles four lanes. Per-lane rates, as listed in the ConnectX-8 manual's IBTA auto-negotiation table:[24]

Note: 4x port speeds are the lane count times the per-lane rate in [24]; NDR and XDR 4x speeds are also stated directly by NVIDIA and the IBTA.[21][25][26] Split ports (HDR100, NDR200) use two lanes.

The IBTA published Volume 1 and 2 Release 2.0 on July 31, 2025, adding XDR at 200 Gb/s per lane for 800 Gb/s (QSFP) and 1.6 Tb/s (OSFP and QSFP-DD) links.[21] Release 2.1, published July 31, 2026, finalizes XDR, adds management support for very high-radix switches, and adds the Multipath Reliable Connection (MRC) specification discussed below.[22][23]

Ethernet for AI: RoCEv2, 400G and 800G

Ethernet's advantages are familiar: multi-vendor supply, standard tooling, one skill set for front-end and back-end, and a very large switch silicon market. Its historic weakness for HPC was transport: TCP is too CPU-heavy and too slow to recover for tightly synchronized collectives. RDMA over Converged Ethernet version 2 (RoCEv2) puts the InfiniBand transport inside routable UDP/IP packets, so NICs can do zero-copy RDMA over a standard IP fabric. The catch, covered in the congestion section, is that classic RoCEv2 expects a lossless network.[38]

Switch silicon has moved quickly. NVIDIA's Spectrum-4 based SN5600 provides 64 ports of 800 GbE and 51.2 Tb/s of switching capacity.[27] Broadcom announced Tomahawk 6 shipping in June 2025 at 102.4 Tb/s with co-packaged optics support,[51] and NVIDIA's Spectrum-6 also doubles per-chip bandwidth to 102.4 Tb/s (128 x 800 Gb/s) using 200G PAM4 SerDes.[10]

Spectrum-X is NVIDIA's name for a tightly coupled Ethernet switch plus SuperNIC platform with adaptive routing, congestion control and telemetry tuned for AI; NVIDIA also describes a "multiplane" option that splits each SuperNIC across two or more independent network planes to scale a two-tier topology.[28] NVIDIA's DGX SuperPOD B300 reference architecture states that Spectrum-X "provides the same 800Gbps connectivity with comparable latency characteristics when compared with XDR InfiniBand."[37] That is a vendor claim, but it reflects the broader trend: large AI buyers now treat tuned Ethernet as a peer of InfiniBand for back-end fabrics.

Ultra Ethernet Consortium: Spec Status

The Ultra Ethernet Consortium (UEC) released Specification 1.0 on June 11, 2025.[29] Point releases followed: 1.0.1 (September 2025), 1.0.2 (January 2026, correcting congestion control algorithms), and 1.0.3 (July 16, 2026), which UEC lists as the current published version.[30] UEC defines a new RDMA transport, Ultra Ethernet Transport (UET), plus link and physical layer enhancements. Its headline features are multipath packet spraying, sender-based congestion control designed for microsecond-scale latency, optional receiver-side credit allocation for incast, and a libfabric-based software interface.[31]

Adoption is early but real. AMD markets its Pensando Pollara 400 NIC as the first UEC-ready AI NIC at 400 Gbps,[52] HPE states it will start introducing UEC compliance in the Slingshot 800 generation,[33] and Slingshot 400 already implements some UEC features such as link-level retry and credit-based flow control on fabric links.[32] Separately, the IBTA's new Multipath Reliable Connection (MRC), contributed jointly by AMD, Broadcom, Intel, Microsoft, NVIDIA and OpenAI, extends the RoCEv2 reliable connection model with multipath, congestion control and failure recovery over standard best-effort Ethernet using existing software interfaces.[22] Expect both paths to coexist for several years.

HPE Slingshot

Slingshot is the fabric in HPE Cray EX systems, including the US exascale machines. It is Ethernet-compliant on the wire but adds HPC-specific extensions.[34] Slingshot 400 pairs a 64-port switch (51.2 Tbps, 64 ports of 400 Gbps) with a 400 Gbps PCIe Gen5 NIC.[32] Its distinguishing features are hardware flow-based congestion management that identifies and throttles the specific flows causing congestion rather than pausing whole ports, and packet-by-packet adaptive routing.[32][33] HPE describes dragonfly and fat-tree networks with over 250,000 endpoints.[33] Software accesses it through the libfabric CXI provider.[34]

RDMA, GPUDirect and DPUs

RDMA Basics

Remote Direct Memory Access lets a NIC read or write another host's registered memory without involving the remote CPU or kernel. The application registers memory, creates queue pairs, and posts work requests to the NIC; completions show up in completion queues. Benefits are low latency, near-zero CPU overhead and line-rate throughput. The InfiniBand verbs API is the common low-level interface for both InfiniBand and RoCE; the Microsoft DCQCN paper reports RDMA latencies well below the TCP stack's on the same hardware and notes that the InfiniBand link layer uses hop-by-hop credit-based flow control to stay lossless.[38]

GPUDirect RDMA

GPUDirect RDMA gives a NIC a direct PCIe path into GPU memory, so network data does not bounce through host RAM. It exposes GPU memory through PCIe BAR windows that a third-party device can read and write like system memory.[35] Practical requirements from NVIDIA's documentation:

• The GPU and NIC should share an upstream PCIe root complex; best performance comes when only PCIe switches sit between them, and paths that cross the CPU socket interconnect can perform badly.[35]
• IOMMU translation must be disabled or set to pass-through for GPUDirect RDMA to work.[35]
• The nvidia-peermem kernel module (added in CUDA 11.4, replacing the out-of-tree nv_peer_mem) provides the peer memory path for NVIDIA networking HCAs; load it with modprobe nvidia-peermem.[35]

NCCL exposes the policy through NCCL_NET_GDR_LEVEL, which sets the maximum PCIe distance at which GPUDirect RDMA is used.[42] This is why server topology matters: modern AI servers pair each GPU with its own NIC under the same PCIe switch. NCCL 2.27 added "Direct NIC" support on Grace Blackwell platforms, attaching CX8 NICs directly to GPUs over PCIe Gen6 x16 to bypass CPU bottlenecks.[41]

Field note: Run nvidia-smi topo -m on every new server model before you accept it. If a GPU and its paired NIC show a path that crosses the CPU interconnect instead of a PCIe switch, your GPUDirect bandwidth will suffer no matter how fast the switches are. Fix it in the BIOS, the riser layout or the NIC-to-GPU mapping, not in NCCL.

DPUs and SuperNICs

NVIDIA now splits its NICs into two roles:

• SuperNICs (ConnectX-8, ConnectX-9) are built for the east-west GPU fabric. ConnectX-8 supports XDR 800 Gb/s InfiniBand on its single-port OSFP models and 400 GbE, with a PCIe Gen6 x16 host interface.[24] ConnectX-9, part of the Rubin platform, is listed at 1,600 Gb/s per GPU.[10]
• DPUs (BlueField-3, BlueField-4) are for infrastructure: storage, security, tenant isolation and north-south traffic. BlueField-3 offers up to 400 Gb/s Ethernet or NDR InfiniBand, up to 16 Arm A78 cores, 16 GB of DDR5 and 32 lanes of PCIe Gen5.[48] BlueField-4 combines a 64-core Grace CPU with ConnectX-9.[10]

A DGX B200 shows the split: eight single-port ConnectX-7 cards (400 Gb/s InfiniBand each) for the compute fabric, and two dual-port BlueField-3 cards for storage and in-band management.[12] For research clusters, DPUs are most useful for multi-tenant isolation and offloaded storage; most sites do not need them to run MPI or NCCL well.

Topologies and Oversubscription

Fat Tree and Clos

The workhorse topology is the folded Clos, usually called a fat tree or leaf-spine. Hosts connect to leaf switches; every leaf connects to every spine. With equal uplink and downlink bandwidth at each leaf the network is non-blocking (1:1): any half of the hosts can talk to the other half at full rate. Switch radix determines scale. A 64-port leaf in a two-tier non-blocking tree uses 32 ports down and 32 up, and 64-port spines can connect 64 leaves, so a two-tier fabric of 64-port switches tops out at 64 x 32 = 2,048 host ports; a third tier multiplies that again. Higher-radix switches (144 ports on Quantum-X800,[26] 128 x 800G on Spectrum-6[10]) flatten the tree, reducing hops, optics and cost.

Rail-Optimized Designs

AI clusters add a twist. In an eight-GPU server, GPU 0 on every node connects to "rail 0," GPU 1 to rail 1, and so on, each rail having its own leaf switch. NCCL arranges collectives so most inter-node traffic flows between same-index GPUs, keeping it one switch hop away; traffic that must change rails can hop over NVLink inside the node first. NVIDIA's DGX SuperPOD B300 reference architecture builds scalable units (SUs) of 64 nodes that are rail-aligned, with traffic per rail "always one hop away from the other 64 nodes in a SU," and a rail-optimized, non-blocking, twin-plane fat tree for the compute fabric.[36][37] The twin-plane design means a plane failure lets a job continue at half bandwidth instead of aborting.[36]

Dragonfly and Torus

Dragonfly groups switches into fully connected groups, then connects every group to every other group with global links. It minimizes long optical cables and hop count, but needs good adaptive routing because minimal paths between two groups share few global links. HPE Cray EX systems use dragonfly throughout.[32][33] NVIDIA also cites DragonFly+ for Quantum-2, claiming over one million 400 Gb/s nodes in a three-hop network.[25]

Torus networks (2D or 3D mesh with wraparound) connect each node to neighbors without a central switch. They are cheap and map well onto nearest-neighbor physics and all-reduce, but bisection bandwidth grows slower than node count, which hurts all-to-all. The TPU v4 paper notes that 2D tori scale bisection bandwidth as N^(1/2) and 3D tori as N^(2/3), and that all-to-all patterns "strain bisection bandwidth."[20]

Oversubscription

Oversubscription is the ratio of downlink to uplink bandwidth at a switch tier. A 2:1 leaf halves worst-case cross-leaf bandwidth. For GPU back-end fabrics running synchronous collectives, the practical answer is: do not oversubscribe. For storage and front-end networks, modest oversubscription is normal; the SuperPOD B300 design connects storage devices at 1:1 but runs the DGX storage connections slightly oversubscribed at roughly 4:3.[36]

Field note: Oversubscribing the GPU compute fabric is the most common false economy in research cluster builds. It looks fine in single-job acceptance tests and then collapses when two large jobs share spine links. If budget forces a compromise, buy fewer nodes with a non-blocking fabric and add nodes later; the SuperPOD guidance is to build the fabric for the full scalable unit and leave ports empty.[36]

Congestion Control, Adaptive Routing and Lossless Ethernet

Lossless vs Lossy

InfiniBand is lossless by design: a sender transmits only when the receiver has advertised buffer credits.[38] Classic RoCEv2 uses a simpler go-back-N recovery that performs badly under loss, so it relies on Priority Flow Control (PFC) to pause an upstream port before buffers overflow. PFC works per port and priority, not per flow, and the DCQCN paper documents what goes wrong: head-of-line blocking, unfairness and congestion spreading, where one hot receiver pauses traffic for innocent flows across the fabric.[38]

DCQCN and What Came After

DCQCN (Microsoft, SIGCOMM 2015) combines switch ECN marking with NIC rate control so that flows slow down before PFC triggers. It became the default RoCE congestion control. Meta's SIGCOMM 2024 paper on its RoCE training clusters is the best public account of running this at scale: Meta found DCQCN hard to tune for training collectives, ran its 400G deployments with PFC only and no transport-level congestion control, and instead added receiver-driven admission control in its collective library, backed by deep-buffer spine switches.[39] The UEC transport and IBTA MRC both take a different approach, designing for multipath and loss tolerance so that the network can run best-effort or near-lossless without fabric-wide PFC.[22][31]

Load Balancing and Adaptive Routing

Standard Ethernet ECMP hashes each flow onto one path using packet header fields. AI training produces few, very large, long-lived flows, so hash collisions leave some links saturated and others idle. Meta reported poor ECMP performance from this low entropy and moved to an enhanced ECMP that also hashes on the RoCE queue pair number, plus centralized traffic engineering, while evaluating flowlet switching.[39] InfiniBand, Spectrum-X and Slingshot use switch-level adaptive routing that picks egress ports per packet or per flowlet based on queue depth; Slingshot applies packet-by-packet adaptive routing to unordered RDMA traffic and flow-level rerouting for ordered traffic.[33] The general lesson: AI traffic needs either adaptive routing in the fabric, multipath in the transport, or careful traffic engineering.

Field note: If you build a RoCE fabric, decide the lossless design on day one and document it: which traffic class is lossless, PFC and ECN thresholds, the DSCP mapping, and the NIC congestion control settings. Then make sure the NIC firmware, switch OS and NCCL (NCCL_IB_GID_INDEX, NCCL_IB_TC) all agree.[42] Most "RoCE is slow" tickets are a single misconfigured switch or a host sending on the wrong priority.

Collectives and Communication Libraries

The Collective Patterns That Matter

NCCL provides AllReduce, Broadcast, Reduce, AllGather, ReduceScatter, AlltoAll, Gather and Scatter, plus point-to-point send and receive.[43] All-to-all can also be built as a grouped loop of ncclSend/ncclRecv calls between ncclGroupStart() and ncclGroupEnd().[57] All-to-all is the hardest pattern on a fabric because it stresses bisection bandwidth rather than neighbor links; Meta's production traffic data showed AlltoAll using far more queue pairs per GPU and more leaf switch buffer than all-reduce.[39] This is a large part of why MoE models push toward bigger scale-up domains like NVL72 and Helios.

NCCL, RCCL, MPI, UCX and libfabric

• NCCL is NVIDIA's topology-aware collective library for GPUs, supporting PCIe, NVLink, InfiniBand verbs and IP sockets.[43] It is configured by environment variables (or /etc/nccl.conf); NVIDIA cautions that debug and tuning variables should be removed once an issue is fixed, since leaving them set can cause sub-optimal behavior or hangs.[42]
• RCCL is AMD's equivalent for Instinct GPUs, providing the same collective primitives over PCIe and xGMI and across nodes.[44]
• MPI remains the standard for scientific codes. MPI 5.0 was approved by the MPI Forum on June 5, 2025,[45] and its headline addition is a standard ABI,[59] which should eventually make containers portable across MPI builds. See Scientific Computing.
• UCX is the transport layer beneath Open MPI and HPC-X on InfiniBand and RoCE. UCX_TLS selects transports (for example rc, ud, dc, sm, cuda_ipc, rocm_ipc) and UCX_NET_DEVICES selects the HCA port.[46]
• libfabric (OFI) is the portable fabric API used by MPICH, Cray MPICH and others, with providers for verbs, TCP, AWS EFA and Slingshot (CXI).[47][34] UEC's software interface is also built on libfabric.[31]

In-Network Computing: SHARP

An all-reduce moves each byte through the network at least twice. SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) lets the switches perform the reduction as data flows up an aggregation tree, so data crosses the network fewer times and GPUs and CPUs do less work.[40] Quantum-2 implements SHARPv3, which supports multiple aggregation trees so many jobs can use it at once.[25][40] Quantum-X800 supports SHARP v4.[26] The same idea inside the NVLink domain is NVLink SHARP (NVLS), where the NVLink Switch performs reductions; NCCL enables it through NCCL_NVLS_ENABLE.[42] NCCL 2.27 extended NVLink and InfiniBand SHARP to AllGather and ReduceScatter, cutting the GPU SMs used for communication from 16 or more to 6 or fewer, according to NVIDIA.[41] UALink 2.0 adds its own In-Network Compute definition.[17]

Optics, Copper and the Physical Layer

Copper vs Optics and Reach

At 100G and 200G per lane, passive copper reach is now measured in a couple of meters. HPE's Slingshot 400 QuickSpecs are a clear example: 400GBASE-CR4 runs up to 1.6 m on passive copper and 5 m on active copper, while 400GBASE-SR4 reaches 100 m on OM4/OM5 multimode fiber.[32] NVIDIA's Spectrum-X800 datasheet lists DAC, linear active copper and optical transceivers, with 800 Gb/s optical links reaching up to 2 km.[27] The practical rule: copper inside the rack (which is why NVL72 and Helios keep the scale-up domain in one rack), active copper or active optical cables to the adjacent rack, and transceivers over fiber for everything else.

Optics Power and Co-Packaged Optics

Pluggable transceivers are a large and growing share of network power and a leading cause of link flaps. NVIDIA's co-packaged optics (CPO) blog cites pluggable interfaces "often" drawing 30 W, versus as low as 9 W when the optical engine sits beside the switch ASIC, and claims 3.5x better power efficiency and 10x better resiliency from fewer discrete components.[50] Treat those as vendor claims. Product status: NVIDIA announced Quantum-X Photonics InfiniBand switches (144 ports of 800 Gb/s, 115 Tb/s, liquid-cooled) and Spectrum-X Photonics Ethernet switches (SN6810 at 128 x 800 Gb/s and SN6800 at 512 x 800 Gb/s), with stated commercial availability of early 2026 for Quantum-X and the second half of 2026 for Spectrum-X.[49][50] Broadcom's Tomahawk 6 also supports CPO.[51] CPO changes serviceability: a failed optical engine is a switch repair, not a transceiver swap, so understand the vendor's field replacement model before buying.

Budget optics as a first-class line item. In a non-blocking two-tier fabric, every host link and every leaf-to-spine link needs a transceiver at each end (or an AOC). On 800G builds the optics can rival the switches in cost and, per link, in power. Keep a spares pool of about the size of your largest rack's worth of transceivers and log every link flap; in our experience flapping optics are a leading source of mysterious NCCL slowdowns.

Front-End, Back-End and Storage Networks

A modern AI/HPC cluster runs several physically or logically separate networks, and mixing them up is a classic design error. Meta describes the split as a front-end (FE) network for storage and data ingestion and a separate non-blocking back-end (BE) RDMA fabric connecting all GPU NICs.[39] NVIDIA's DGX SuperPOD B300 reference architecture uses four fabrics:[36][37]

• Compute (back-end) fabric: the GPU-to-GPU RDMA network, rail-optimized and non-blocking, 800 Gb/s XDR InfiniBand or Spectrum-X Ethernet in the B300 design.
• Storage fabric: RDMA (InfiniBand or RoCE) to the parallel file system, independent of the compute fabric so checkpoint writes do not collide with collectives.
• In-band management (front-end): Ethernet for provisioning, Slurm and Kubernetes services, user logins, home directories over NFS and internet access. Compute nodes connect at 200 Gbps, bonded, in that design; user storage needs at least 100 Gb/s.
• Out-of-band management: 1 GbE for BMCs, PDUs and switch management, isolated from users.

Smaller research clusters often merge the storage and compute fabrics onto one InfiniBand network, which is reasonable if the fabric is non-blocking and storage traffic is separated with InfiniBand partitions or service levels. Do not merge the back-end GPU fabric with the general-purpose campus-facing Ethernet. See Memory and Storage for file system sizing.

Practical Testing

Test from the bottom up: link state, then raw RDMA per NIC, then MPI, then NCCL collectives inside one node, then across nodes. Record baselines at acceptance so you have something to compare against later (see Observability).

Link and Topology Checks

# Link state, rate and firmware for each HCA port
ibstat
ibv_devinfo

# GPU-to-GPU and GPU-to-NIC topology (look for PIX/PXB between each GPU and its NIC)
nvidia-smi topo -m

# Confirm the GPUDirect RDMA peer memory module is loaded
lsmod | grep nvidia_peermem || sudo modprobe nvidia-peermem

Raw RDMA Bandwidth with perftest

The linux-rdma perftest suite (ib_write_bw, ib_read_bw, ib_send_bw, and the matching _lat tests) measures what one NIC can do with no MPI or NCCL in the way. Build it with CUDA support to test GPUDirect RDMA using --use_cuda.[53] The flags below are documented in perftest's README and vendor guides: -d device, -x GID index (needed for RoCE), -F suppress CPU frequency warnings, --report_gbits, -a sweep all message sizes, --use_cuda GPU index.[54]

# Build with CUDA support (example paths)
git clone https://github.com/linux-rdma/perftest.git
cd perftest
./autogen.sh
./configure CUDA_H_PATH=/usr/local/cuda/include/cuda.h
make -j

# Server (node A): host memory, then GPU 0 memory
./ib_write_bw -d mlx5_0 -F --report_gbits -a
./ib_write_bw -d mlx5_0 -F --report_gbits -a --use_cuda=0

# Client (node B): same flags plus the server's hostname
./ib_write_bw -d mlx5_0 -F --report_gbits -a --use_cuda=0 nodeA

# RoCE: add the GID index for RoCEv2 (example value; check with show_gids)
./ib_write_bw -d mlx5_0 -x 3 -F --report_gbits -a nodeA

Large-message results should approach the port rate. If host-memory results are good but --use_cuda results are poor, suspect PCIe topology, IOMMU settings or a missing peer memory module.[35]

MPI with the OSU Micro-Benchmarks

The OSU Micro-Benchmarks (OMB) from Ohio State cover point-to-point latency and bandwidth plus blocking and non-blocking collectives; OMB 7.5.2 is the current release with 8.0rc available.[55] Built with --enable-cuda, the D D arguments place send and receive buffers in GPU memory.[56]

# Two nodes, one rank each, Open MPI over UCX, GPU buffers on both sides
mpirun -np 2 -npernode 1 -H nodeA,nodeB \
  -mca pml ucx \
  -x UCX_NET_DEVICES=mlx5_0:1 \
  -x CUDA_VISIBLE_DEVICES=0 \
  ./osu_bw D D

# Small-message latency, host buffers (classic MPI health check)
mpirun -np 2 -npernode 1 -H nodeA,nodeB -mca pml ucx ./osu_latency

# Collective latency across all ranks
mpirun -np 64 -npernode 8 -mca pml ucx ./osu_allreduce

Collectives with nccl-tests

nccl-tests is the standard acceptance test for GPU fabrics. Build with MPI=1 for multi-node runs. The README's own examples: a single node with 8 GPUs, and 64 ranks across 8 nodes.[2]

make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl

# One node, 8 GPUs: tests NVLink/NVSwitch (or xGMI with rccl-tests)
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8

# 8 nodes x 8 GPUs = 64 ranks: tests the scale-out fabric
mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

A Slurm version for a two-node smoke test (illustrative values; adjust partition, modules and HCA names to your site):

#!/bin/bash
#SBATCH --job-name=nccl-allreduce
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=00:20:00

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET
export NCCL_IB_HCA=mlx5          # example: restrict NCCL to mlx5 RDMA devices
export NCCL_SOCKET_IFNAME=ib0    # example: interface for NCCL bootstrap traffic

srun --mpi=pmix ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
srun --mpi=pmix ./build/alltoall_perf   -b 8 -e 1G -f 2 -g 1

Read the busbw column, not algbw. For large messages across nodes, busbw should approach the per-GPU NIC bandwidth; inside an NVLink domain it should approach the NVLink figure.[1] NCCL_DEBUG=INFO prints which transports and NICs NCCL chose; check that every rank reports the RDMA NET transport and GPUDirect RDMA where expected. NCCL_TOPO_DUMP_FILE writes the detected topology to a file for inspection.[42]

Field note: Run nccl-tests on every node pair or rack pair after any hardware change, not just at acceptance. A single degraded link (a port that negotiated down, a dirty fiber, a NIC with an old firmware) will show up as one node pair with low busbw, and it will slow every job that lands on that node. Automate it as a scheduled health check (see Operations).

Design Guidance for Research Clusters

These are starting points, not prescriptions. The right design depends on workload mix, budget, power and staff skills. Product examples are illustrative, not endorsements.

8 GPUs: One Node

• Buy a server with an NVLink/NVSwitch (HGX) or Infinity Fabric baseboard rather than PCIe-only GPUs if you expect multi-GPU training. The scale-up domain is the whole machine, so it is the only interconnect that matters for GPU-to-GPU traffic.[8][13]
• The external network is for storage and users. One or two 100-400 Gb/s ports to the storage and campus network are usually enough. Skip the back-end GPU fabric entirely.
• Validate with single-node all_reduce_perf -g 8 and nvidia-smi topo -m.[2]

64 GPUs: Eight Nodes

• This is the point where a dedicated back-end RDMA fabric pays for itself. Give each GPU its own NIC (eight per node), as in DGX-class designs.[12]
• Eight nodes x eight NICs is 64 ports, which fits on a single 64-port NDR switch (Quantum-2 class)[25] or a single 800G Ethernet switch (SN5600 class)[27] with no oversubscription and one hop between any two GPUs. A single high-radix switch is simpler than a rail-optimized multi-switch design at this size and leaves room for storage ports on a separate switch.
• InfiniBand vs Ethernet: InfiniBand is still the lower-effort path for a small team (subnet manager, lossless by default, SHARP, mature MPI stack). Choose Ethernet if your staff already runs RoCE, your vendor provides a validated AI Ethernet configuration, or you must converge with existing Ethernet tooling.
• Keep storage on a separate network or partition so checkpoints do not collide with collectives.

512+ GPUs: Tens of Nodes and Beyond

• Use a rail-optimized, non-blocking two-tier fat tree built in scalable units. NVIDIA's reference design uses 64-node SUs (512 GPUs) with rails one hop apart inside the SU.[36] Design the spine for the final size on day one.
• Consider multi-plane designs (twin-plane InfiniBand, Spectrum-X multiplane, multiplane Ethernet with packet-spraying NICs) for resilience and to stay at two tiers as you grow.[28][36]
• Decide congestion control and load balancing early: InfiniBand adaptive routing and SHARP, or an Ethernet platform with adaptive routing or multipath transport. Plain ECMP with default RoCE settings is not adequate for large collectives.[39]
• If your workloads are MoE training or large-model inference, evaluate rack-scale NVLink (NVL72) or UALink (Helios) domains. They move all-to-all traffic off the scale-out fabric.[9][14] The facility cost is high: liquid cooling and very high rack power.
• Staff for it. A fabric this size needs a subnet manager or fabric manager strategy, telemetry, automated link health checks and optics spares. Integrate topology into the scheduler so jobs land on nearby nodes (see Scheduling).

Key Takeaways

• Collectives finish only when the slowest link finishes; the interconnect sets the pace of distributed training and tightly coupled simulation. Benchmark across message sizes, because small messages are latency-bound and large ones are bandwidth-bound.
• Scale-up fabrics (NVLink, Infinity Fabric, UALink, TPU ICI) carry an order of magnitude more bandwidth per accelerator than scale-out NICs. Put tensor and expert parallelism inside the scale-up domain.
• Rack-scale domains (GB300 and Vera Rubin NVL72, AMD Helios with UALink over Ethernet) are the big change since the last edition, driven by MoE all-to-all traffic.
• InfiniBand XDR (800 Gb/s per port) and 800G Ethernet are the current scale-out generation; UEC 1.0.3 and IBTA MRC are bringing multipath, loss-tolerant RDMA to standard Ethernet.
• GPUDirect RDMA depends on PCIe topology: pair each GPU with its own NIC under a common PCIe switch and verify with nvidia-smi topo -m.
• Do not oversubscribe the GPU back-end fabric. Keep compute, storage, in-band and out-of-band networks distinct.
• Test bottom up (ibstat, ib_write_bw, OSU, nccl-tests), record baselines at acceptance, and rerun them as automated health checks.

Sources

1. nccl-tests Performance documentation: algbw and busbw (NVIDIA, GitHub, 2026)
2. nccl-tests README (NVIDIA, GitHub, 2026)
3. What bit rates does the PCIe 5.0 specification support? (PCI-SIG FAQ)
4. PCI Express 6.0 Specification (PCI-SIG)
5. The PCIe 7.0 Specification, Version 1.0 is Now Available to Members (PCI-SIG, 2025)
6. PCIe 7.0 Specification, Version 0.5 Now Available (PCI-SIG, 2024)
7. PCIe 8.0 Specification, Draft 0.5 Now Available (PCI-SIG, 2026)
8. NVLink and NVLink Switch (NVIDIA, 2026)
9. NVIDIA GB300 NVL72 (NVIDIA, 2026)
10. Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer (NVIDIA Technical Blog, 2026)
11. NVIDIA Kicks Off the Next Generation of AI With Rubin (NVIDIA Newsroom, January 2026)
12. Introduction to NVIDIA DGX B200 Systems (NVIDIA DGX B200 User Guide, 2026)
13. AMD Instinct MI355X GPUs specifications (AMD)
14. AMD Helios: Resilient Scale-Up Networking for AI (AMD, 2026)
15. AMD Launches Helios rackscale AI infrastructure (AMD, July 2026)
16. UALink Consortium Releases the Ultra Accelerator Link 200G 1.0 Specification (UALink Consortium, April 2025)
17. UALink Consortium Publishes Four Specifications Defining In-Network Compute, Chiplets, Manageability and 200G Performance (UALink Consortium, April 2026)
18. UALink Specifications (UALink Consortium, 2026)
19. TPU7x (Ironwood) documentation (Google Cloud, 2026)
20. Jouppi et al., TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (ISCA 2023, arXiv)
21. IBTA Advances High-Performance Networking with New Specification Updates, Release 2.0 (IBTA, 2025)
22. RDMA Networking Powers the June 2026 TOP500 while IBTA Advances New Specifications (IBTA, 2026)
23. What's New in InfiniBand Volume 1 and 2 Release 2.1 (IBTA, July 2026)
24. NVIDIA ConnectX-8 SuperNIC User Manual: Specifications (NVIDIA, 2026)
25. NVIDIA Quantum-2 InfiniBand Platform (NVIDIA)
26. NVIDIA Quantum-X800 InfiniBand Platform (NVIDIA)
27. NVIDIA Spectrum-X800 Ethernet Platform Datasheet (NVIDIA)
28. NVIDIA Spectrum-X Ethernet Networking Platform (NVIDIA, 2026)
29. Ultra Ethernet Consortium Launches Specification 1.0 (UEC, June 2025)
30. Ultra Ethernet Specification History (UEC, 2026)
31. UEC 1.0 white paper (Intersect360 Research for UEC, 2025)
32. HPE Slingshot interconnect 400 QuickSpecs (HPE)
33. The HPE Slingshot 400 Expedition (Cray User Group, May 2025)
34. fi_cxi(7): The CXI Fabric Provider (libfabric)
35. GPUDirect RDMA documentation (NVIDIA CUDA 13.4)
36. DGX SuperPOD B300 Reference Architecture: Network Fabrics (NVIDIA, 2026)
37. DGX SuperPOD B300 Reference Architecture: Key Components (NVIDIA, 2026)
38. Zhu et al., Congestion Control for Large-Scale RDMA Deployments (SIGCOMM 2015)
39. Gangidi et al., RDMA over Ethernet for Distributed AI Training at Meta Scale (SIGCOMM 2024)
40. NVIDIA SHARP Rev 3.16.0: Introduction (NVIDIA)
41. Enabling Fast Inference and Resilient Training with NCCL 2.27 (NVIDIA Technical Blog, 2025)
42. NCCL Environment Variables (NVIDIA NCCL documentation)
43. Overview of NCCL (NVIDIA NCCL documentation)
44. RCCL documentation (AMD ROCm)
45. MPI Documents: MPI 5.0 (MPI Forum)
46. UCX environment parameters (OpenUCX wiki)
47. Libfabric (OpenFabrics Interfaces Working Group)
48. NVIDIA BlueField-3 DPU Datasheet (NVIDIA)
49. NVIDIA Announces Spectrum-X Photonics, Co-Packaged Optics Networking Switches (NVIDIA Newsroom, 2025)
50. Scaling AI Factories with Co-Packaged Optics for Better Power Efficiency (NVIDIA Technical Blog, August 2025)
51. Broadcom Ships Tomahawk 6: World's First 102.4 Tbps Switch (Broadcom, June 2025)
52. AMD Pensando Pollara 400 AI NIC (AMD)
53. perftest README (linux-rdma, GitHub)
54. Testing RDMA (perftest) (Broadcom TechDocs)
55. OSU Micro-Benchmarks (MVAPICH, Ohio State University, 2026)
56. NVIDIA GPUDirect RDMA User Manual: Benchmark Tests (NVIDIA)
57. NCCL Point-to-point communication (NVIDIA NCCL documentation)
58. AAI 2026: AMD Launches AMD Helios Rackscale Solution for Frontier AI (AMD Newsroom, July 2026)
59. The New MPI 5.0 - Now with ABI Included! BoF at ISC25 (MPI Forum, June 2025)
60. NVIDIA Vera Rubin NVL72 product page (NVIDIA, 2026)