Glossary

AI and HPC grew up with separate vocabularies, and the convergence has left research computing staff fluent in one dialect and guessing in the other. This glossary puts both in one place. Definitions are short and factual; where a term has a home chapter in this manual, the entry links to it. Hard numbers are cited to primary sources and reflect what was published as of September 2026. Terms are grouped by their first letter; acronyms are listed under the letter they start with.

A

Accelerator: A processor specialized for a class of computation, most commonly a GPU, that sits alongside a general-purpose CPU host and does the bulk of the floating-point work. See Accelerators.

Activation checkpointing: A training technique that discards intermediate activations during the forward pass and recomputes them during the backward pass, trading extra compute for lower memory use. The Megatron-LM paper lists it among the standard memory management techniques for large models.[8] See Training at Scale.

Agentic workflow: A workflow in which an LLM-driven agent plans steps, calls tools, launches jobs and acts on the results with limited human intervention. MLPerf Inference v6.1 added an agentic benchmark to reflect this workload.[41] See The Convergence.

All-gather: A collective operation in which each of k ranks contributes N values and every rank receives the full k x N result, ordered by rank.[2]

All-reduce: A collective operation that combines (for example, sums) arrays from all ranks and leaves the identical result on every rank. It is the core operation of data-parallel training, and it is equivalent to a reduce-scatter followed by an all-gather.[2] See Interconnects.

All-to-all: A collective in which every rank sends a distinct piece of data to every other rank. It dominates communication in Mixture-of-Experts models, where tokens are routed to experts on other GPUs; NCCL provides it as a primitive.[1]

APU (Accelerated Processing Unit): A package that combines CPU cores and GPU compute on the same device with shared memory. El Capitan's nodes each hold four AMD MI300A APUs with 128 GB of memory per APU.[38]

Apptainer: An open-source container platform, first developed at Lawrence Berkeley National Laboratory, designed to run applications on HPC clusters as single-file images that integrate with GPUs, high-speed networks and parallel file systems by default.[19] See Systems Software.

Arithmetic intensity: The number of floating-point operations performed per byte moved from memory. It determines whether a kernel is memory-bound or compute-bound on a given processor.[31] See The Convergence.

B

Backfill scheduling: A scheduler policy that starts smaller or shorter jobs out of priority order when doing so will not delay the expected start of higher-priority jobs. Slurm offers it as an optional plugin.[17] See Scheduling.

BF16 (bfloat16): A 16-bit floating-point format with 1 sign bit, 8 exponent bits and 7 mantissa bits, giving it the same dynamic range as FP32 at lower precision. It was designed at Google Brain for TPUs and is the default AI training format today.[35]

Bisection bandwidth: The total bandwidth across the narrowest cut that splits a network into two equal halves; a measure of how well a fabric handles all-to-all traffic. Aurora, for example, lists a peak bisection bandwidth of 0.69 PB/s.[39]

C

Checkpoint: A saved snapshot of a running job's state (model weights and optimizer state, or simulation fields) that allows restart after failure or preemption. Large training checkpoints are a major storage workload. See Training at Scale and Memory and Storage.

Collective (collective operation): A communication operation in which every rank in a group participates, such as broadcast, reduce, all-reduce, all-gather and reduce-scatter. Each rank must call the collective with the same count and datatype; mismatches can cause hangs or data corruption.[2]

Continuous batching: An inference serving technique that adds new requests to the running batch as soon as earlier requests finish, instead of waiting for a whole batch to complete. vLLM lists it among its core features.[5] See Inference at Scale.

CUDA: NVIDIA's parallel computing platform and programming model for its GPUs, usable from C, C++ and Fortran.[21] See Systems Software.

CXL (Compute Express Link): An open interconnect standard built on the PCIe physical layer for cache-coherent attachment of memory and accelerators. CXL 4.0 is based on PCIe 7.0 and doubles the data rate to 128 GT/s.[26] See Memory and Storage.

D

DAOS (Distributed Asynchronous Object Storage): An open-source object store designed for massively distributed non-volatile memory and NVMe, presenting a key-value interface on commodity hardware.[42] Aurora's 230 PB DAOS system is rated at 31 TB/s.[39]

Data parallelism: A training strategy in which every GPU holds a full copy of the model, processes a different slice of each batch, and synchronizes gradients with an all-reduce. It is the simplest form of distributed training. See Training at Scale.

Decode (decode phase): The token-by-token generation phase of LLM inference, in which each step reads the model weights and KV cache to produce one new token per sequence. It is typically memory-bandwidth-bound.[31] See Inference at Scale.

Direct liquid cooling (DLC): Cooling in which liquid flows through cold plates attached to processors and other hot components, rather than relying on air. JUPITER Booster and the June 2026 Green500 leaders use direct liquid-cooled BullSequana XH3000 hardware.[37] See Facilities.

DPU (Data Processing Unit): A programmable network adapter with its own processor cores that offloads networking, storage and security functions from the host. NVIDIA describes its BlueField DPUs as enabling cloud networking, composable storage, zero-trust security and GPU compute elasticity.[40]

Dragonfly: A hierarchical network topology in which groups of switches are densely connected internally and each group has direct links to other groups, reducing the number of hops across a large system. Aurora's Slingshot 11 network uses a dragonfly topology with adaptive routing.[39] See Interconnects.

E

Emulated FP64: Computing FP64-accurate results, typically matrix multiplications, on lower-precision hardware units by splitting operands into slices and combining the partial products. cuBLAS in CUDA 13.0 Update 2 added FP64 matmul emulation on Blackwell INT8 tensor cores.[29] See Scientific Computing.

Exascale: Sustained performance of at least 10^18 floating-point operations per second, conventionally measured in FP64 on HPL. Frontier was the first system to cross it on HPL, at 1.102 Exaflop/s in May 2022.[43] Five systems exceed it on the June 2026 TOP500.[37]

Expert parallelism: A parallelism strategy for Mixture-of-Experts models that places different experts on different GPUs and routes tokens to them with all-to-all communication. See Training at Scale.

F

Fat tree: A multi-level switched network topology in which link capacity increases toward the core so that, when fully provisioned, any half of the nodes can talk to the other half at full bandwidth. It is the most common InfiniBand cluster topology. See Interconnects.

Fine-tuning: Continuing to train a pre-trained model on a smaller, task-specific dataset. Full fine-tuning updates all parameters; parameter-efficient methods such as LoRA update a small subset. See Training at Scale.

FlashAttention: An exact attention algorithm that uses tiling to reduce reads and writes between GPU HBM and on-chip SRAM, making attention faster and more memory-efficient without approximation.[11]

FLOPS: Floating-point operations per second, the standard measure of compute throughput. A FLOPS figure is meaningless without the precision (FP64, BF16, FP4 and so on) and whether it assumes sparsity. See The Convergence.

FP4: A 4-bit floating-point format; the common E2M1 layout has 1 sign bit, 2 exponent bits and 1 mantissa bit and represents values from about -6 to 6. It is usable in practice only with block scaling, as in MXFP4 or NVFP4.[33]

FP8: An 8-bit floating-point format with two encodings: E4M3 (4 exponent, 3 mantissa bits) and E5M2 (5 exponent, 2 mantissa bits). FP8 training has been shown to match 16-bit results on models up to 175 billion parameters.[34]

FP16 (half precision): The IEEE 754 16-bit format, with 5 exponent bits and 10 mantissa bits. Its narrow range made loss scaling necessary in early mixed-precision training.[36][34]

FP32 (single precision): The IEEE 754 32-bit format, with 8 exponent bits and 23 stored mantissa bits. It is used for many science codes and for master weights in AI training. See The Convergence.

FP64 (double precision): The IEEE 754 64-bit format, with a 53-bit significand.[29] It is the precision used by HPL and required by many simulation codes for stability and validated results.

FSDP (Fully Sharded Data Parallel): PyTorch's implementation of sharded data parallelism, which splits parameters, gradients and optimizer state across data-parallel workers. The PyTorch documentation describes it as inspired by ZeRO Stage 3.[7] See Training at Scale.

G

GEMM (General Matrix Multiply): The BLAS level-3 operation C = alpha x A x B + beta x C. Tensor cores and matrix units are built to accelerate it, and it dominates both transformer training and dense linear algebra; DGEMM is the FP64 version.

GPU (Graphics Processing Unit): A massively parallel processor, originally for graphics, that is now the dominant accelerator for both AI and HPC. See Accelerators.

GPUDirect RDMA: An NVIDIA technology, introduced with Kepler-class GPUs and CUDA 5.0, that gives third-party PCIe devices such as network adapters a direct path to GPU memory without staging through host memory. The two devices generally must share the same upstream PCIe root complex.[3] See Interconnects.

Gradient accumulation: Summing gradients over several micro-batches before an optimizer step, which gives the effect of a larger batch without the memory cost. See Training at Scale.

Green500: The TOP500's companion ranking by energy efficiency (GFLOPS per watt on HPL). In June 2026 the leader is KAIROS at the University of Toulouse at 73.28 GFLOPS/W.[37] See Facilities.

H

HBM (High Bandwidth Memory): Stacked DRAM placed next to the processor on the same package, with a very wide interface. The JEDEC HBM4 standard (JESD270-4, April 2025) defines a 2048-bit interface at up to 8 Gb/s for up to 2 TB/s, and stacks up to 64 GB.[27] See Memory and Storage.

HIP: AMD's C++ GPU programming interface, part of the ROCm stack, designed so that CUDA-style code can be ported to AMD GPUs.[20]

HPCG (High Performance Conjugate Gradients): A benchmark that measures performance on sparse, memory-bound access patterns closer to real solvers than HPL. June 2026 scores are roughly one percent of the leading systems' HPL scores.[37]

HPL (High Performance Linpack): The benchmark used to rank the TOP500: solving a dense linear system in FP64. It rewards peak FP64 matrix throughput and a well-tuned network.[37]

HPL-MxP: A mixed-precision variant of HPL (formerly HPL-AI) that performs the LU factorization in low precision and uses iterative refinement to recover FP64 accuracy.[30] El Capitan leads the June 2026 list at 16.7 EF/s.[37]

I

InfiniBand: An industry-standard switched fabric maintained by the InfiniBand Trade Association, providing reliable messaging and RDMA without software in the data path.[22] See Interconnects.

Inference: Running a trained model to produce outputs (predictions, generated tokens) for new inputs. At scale, inference is a serving problem dominated by latency, memory bandwidth and KV cache capacity. See Inference at Scale.

Iterative refinement: A numerical technique that solves a system in low precision, then repeatedly computes the residual in high precision and corrects the solution until it reaches the desired accuracy. It underlies HPL-MxP.[30]

K

KV cache (key-value cache): The stored attention keys and values for all previous tokens of each sequence during LLM inference, which avoids recomputing them at every step. It is large and grows and shrinks dynamically, which makes its memory management a central serving problem.[4] See Inference at Scale.

Kubernetes: A portable, extensible, open-source platform for managing containerized workloads and services, open-sourced by Google in 2014.[18] In research computing it is used mainly for serving and services alongside batch schedulers. See Scheduling.

L

LLM (Large Language Model): A neural network, almost always transformer-based, trained on large text corpora to predict the next token. See Training and Inference.

LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning method that freezes the pre-trained weights and trains small low-rank matrices injected into each layer. For GPT-3 175B, the authors report 10,000x fewer trainable parameters and 3x lower GPU memory than full fine-tuning with Adam.[12]

Lustre: An open-source parallel file system widely used on leadership-class HPC systems.[44] See Memory and Storage.

M

Memory wall: The growing gap between processor compute throughput and memory bandwidth, a term coined by Wulf and McKee in 1995. Gholami and colleagues report peak server FLOPS growing 3.0x every two years versus 1.6x for DRAM bandwidth.[31]

MIG (Multi-Instance GPU): An NVIDIA feature, starting with the Ampere architecture, that partitions one GPU into up to seven isolated instances, each with dedicated compute, L2 cache and memory paths.[15] See Scheduling.

Mixed precision: Using lower-precision formats for most arithmetic while keeping selected values (master weights, accumulations) in higher precision. The original recipe keeps an FP32 weight copy, applies loss scaling and accumulates half-precision products into FP32.[36]

MLPerf: The benchmark suites run by the MLCommons consortium that measure AI training time-to-accuracy and inference throughput on fixed reference models. MLPerf began in 2018.[32]

MoE (Mixture of Experts): A model architecture in which a trainable gating network routes each input to a small subset of many expert sub-networks, so total parameters can be very large while per-token compute stays modest.[10] DeepSeek V3, an MLPerf Training benchmark, has 671 billion total parameters with 37 billion active per token.[32]

MPI (Message Passing Interface): The standard API for message passing between processes in parallel programs, maintained by the MPI Forum. MPI 5.0 was approved on June 5, 2025.[25] See Scientific Computing.

MPS (Multi-Process Service): An NVIDIA facility that lets multiple processes share one GPU concurrently through a common server process, improving utilization for small jobs without MIG's hardware partitioning. See Scheduling.

MX (Microscaling) formats: Narrow data formats in which a small block of values shares one scale factor. The OCP MX specification v1.0 defines MXFP8, MXFP6, MXFP4 and MXINT8; MXFP4 uses a power-of-two scale per 32-value block.[16][33]

N

NCCL (NVIDIA Collective Communications Library): A topology-aware library of GPU collective and point-to-point communication primitives that runs over PCIe, NVLink, InfiniBand verbs and IP sockets. Its API closely follows MPI collectives, with an added CUDA stream argument.[1] See Interconnects.

NUMA (Non-Uniform Memory Access): A memory architecture in which access time depends on which socket or device the memory is attached to. On El Capitan nodes, each APU socket with its 128 GB defines a NUMA domain.[38]

NVFP4: NVIDIA's 4-bit floating-point format for Blackwell and later GPUs: E2M1 values with an FP8 (E4M3) scale per 16-value block plus an FP32 per-tensor scale, about 4.5 bits per value.[33]

NVLink: NVIDIA's high-bandwidth GPU-to-GPU interconnect for scale-up. Fifth-generation NVLink (Blackwell) provides 1,800 GB/s per GPU; sixth-generation (Rubin, preliminary specification) lists 3,600 GB/s.[28] See Interconnects.

NVSwitch (NVLink Switch): The switch chip that connects many NVLinks so that every GPU in a domain can talk to every other at full NVLink speed. In an NVL72 rack it connects 72 GPUs in an all-to-all topology, and it includes SHARP engines for in-network reductions.[28]

O

OpenMP: A directive-based API for shared-memory parallel programming in C, C++ and Fortran, now also used to offload loops to GPUs. It is supported in the ROCm and oneAPI stacks.[20][39]

Ozaki scheme: An accurate matrix multiplication method, proposed by Ozaki and colleagues in 2012, that computes FP64 matrix products using lower-precision units such as FP16, INT8 or FP8 tensor cores.[24] See Scientific Computing.

P

PagedAttention: An attention algorithm, inspired by virtual memory paging in operating systems, that stores the KV cache in non-contiguous blocks to nearly eliminate memory waste. The vLLM authors report 2-4x higher throughput than prior systems at the same latency.[4]

Parallel file system: A file system that stripes data across many storage servers so that many clients can read and write concurrently at high aggregate bandwidth. Examples include Lustre, IBM Storage Scale (GPFS) and DAOS. See Memory and Storage.

PEFT (Parameter-Efficient Fine-Tuning): The family of fine-tuning methods, including LoRA, that train only a small fraction of a model's parameters. See Training at Scale.

Pipeline parallelism: A model-parallel strategy that places consecutive groups of layers on different accelerators and streams micro-batches through them. GPipe introduced it as a general library for any network expressible as a sequence of layers.[9]

Prefill: The first phase of LLM inference, in which the whole input prompt is processed in parallel to populate the KV cache. It is typically compute-bound, unlike decode; vLLM supports chunked prefill to interleave it with decoding.[5]

PUE (Power Usage Effectiveness): Total facility power divided by IT equipment power. A value of 1.0 would mean every watt goes to computing. See Facilities.

Q

Quantization: Converting model weights and/or activations to a lower-precision format (INT8, FP8, FP4) to reduce memory footprint and increase throughput. NVIDIA reports 1 percent or less accuracy loss when quantizing DeepSeek-R1-0528 from FP8 to NVFP4.[33] See Inference at Scale.

R

RAG (Retrieval-Augmented Generation): An LLM application pattern that retrieves relevant documents from a search index or vector database and adds them to the prompt. MLPerf Inference v6.1 added an end-to-end RAG benchmark.[41]

RCCL: AMD's ROCm collective communication library, the counterpart to NCCL for AMD GPUs. See Interconnects.

RDMA (Remote Direct Memory Access): Moving data directly between the memory of two machines through the network adapters, without involving either host's operating system in the data path. It is the basis of InfiniBand and RoCE.[22]

Reduce-scatter: A collective that performs a reduction and then splits the result into equal blocks, one per rank. It is the first half of a ring all-reduce and a key operation in sharded training.[2]

Ridge point: On a roofline plot, the arithmetic intensity at which a processor shifts from memory-bound to compute-bound, equal to peak FLOPS divided by memory bandwidth. See The Convergence.

RoCE (RDMA over Converged Ethernet): A protocol that carries RDMA over Ethernet networks. Its specifications are maintained by the InfiniBand Trade Association.[23] See Interconnects.

ROCm: AMD's mostly open-source software stack for programming AMD GPUs, providing compilers, libraries, debuggers, profilers and runtimes for HIP, OpenCL and OpenMP.[20] See Systems Software.

Roofline model: A visual performance model, introduced by Williams, Waterman and Patterson, that bounds attainable performance by peak compute and by memory bandwidth times arithmetic intensity.[6] See The Convergence.

S

Scale-out: Adding capacity by connecting more nodes over a network such as InfiniBand or Ethernet. See Interconnects.

Scale-up: Adding capacity within a single tightly coupled domain, such as GPUs joined by NVLink inside a node or rack, where bandwidth is much higher and latency much lower than on the scale-out network. See Interconnects.

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol): NVIDIA's in-network computing technology that performs reductions inside switches rather than on endpoints. NVLink Switch chips include SHARP engines for in-network reductions and multicast.[28]

Slingshot: HPE Cray's high-performance interconnect used on El Capitan, Frontier and Aurora. On Aurora, each Slingshot switch provides 25.6 Tb/s across 64 ports of 200 Gb/s.[39]

Slurm: An open-source cluster management and job scheduling system for Linux clusters. A central slurmctld daemon manages resources and the queue, and a slurmd daemon on each node runs work.[17] See Scheduling.

Sparsity (structured sparsity): Hardware support for skipping zero values in a fixed pattern of weights. NVIDIA introduced it with Ampere, doubling tensor core throughput for structured sparse matrices, which is why vendor spec sheets often show separate "sparse" and "dense" figures.[14]

Speculative decoding: An inference technique in which a small draft model proposes several tokens that the large model verifies in parallel, producing identical output distributions faster. The original paper reports 2-3x acceleration on T5-XXL.[13]

Surrogate model: A machine-learned model trained to approximate an expensive simulation or physical process. GraphCast, trained on reanalysis data, produces 10-day global weather forecasts at 0.25 degree resolution in under one minute.[45] See Scientific Computing.

Systolic array: A grid of multiply-accumulate units through which data flows rhythmically, each unit passing partial results to its neighbor. Google's TPUs use systolic arrays to accelerate matrix multiplication.[35]

T

Tensor core: NVIDIA's name for the matrix-multiply-accumulate units in its GPUs; AMD calls the equivalent matrix cores. Each generation adds formats: Ampere's third-generation tensor cores added TF32, BF16 and FP64 support.[14] See Accelerators.

Tensor parallelism: A model-parallel strategy that splits individual layers (weight matrices) across GPUs, requiring communication inside each layer. Megatron-LM introduced the widely used intra-layer approach for transformers.[8] See Training at Scale.

TF32 (TensorFloat-32): An NVIDIA tensor core math mode with 8 exponent bits (FP32 range) and 10 mantissa bits (FP16 precision). It reads and writes FP32 data, so it can accelerate FP32 code with no data format changes.[14]

TFLOPS / PFLOPS / EFLOPS: 10^12, 10^15 and 10^18 floating-point operations per second. Always check the precision and sparsity assumption behind the number. See Accelerators.

Token: The unit of text (a word piece, character group or symbol) that a language model reads and generates. Inference throughput is reported in tokens per second.

TOP500: The twice-yearly (June and November) ranking of the world's most powerful supercomputers by HPL, begun in 1993.[37] See The Convergence.

TPU (Tensor Processing Unit): Google's custom AI accelerator, built around systolic arrays and the bfloat16 format, available through Google Cloud.[35] See Accelerators and Cloud.

Transformer: The neural network architecture, introduced in 2017, built entirely on attention mechanisms without recurrence or convolution. It underlies nearly all current LLMs.[46]

U

UALink (Ultra Accelerator Link): An open scale-up interconnect standard for accelerator-to-accelerator and accelerator-to-switch communication. The UALink 200G 1.0 specification defines 200G per lane connections for up to 1,024 accelerators in a pod.[49] See Interconnects.

Ultra Ethernet (UEC): An Ethernet-based communication stack for AI and HPC defined by the Ultra Ethernet Consortium, covering transport, congestion control and RDMA across NICs, switches, optics and cables. Specification 1.0 was released on June 11, 2025.[48] See Interconnects.

V

vLLM: An open-source LLM inference and serving engine, originally developed at UC Berkeley's Sky Computing Lab, built around PagedAttention, continuous batching and prefix caching, with an OpenAI-compatible API server and support for NVIDIA and AMD GPUs.[5] See Inference at Scale.

X

XDR InfiniBand: The InfiniBand generation defined in IBTA Volume 1 Release 1.7 (October 2023), with 200 Gb/s per lane SerDes, 800 Gb/s per port and 1.6 Tb/s switch-to-switch links. The preceding NDR generation runs at 400 Gb/s.[23][22]

Z

ZeRO (Zero Redundancy Optimizer): A family of memory optimizations from Microsoft DeepSpeed that eliminates redundant copies of optimizer state, gradients and parameters across data-parallel workers, in progressive stages. Its authors trained models over 100 billion parameters on 400 GPUs.[47] See Training at Scale.

Key Takeaways

Many "new" AI terms map directly onto old HPC ideas: collectives are MPI collectives, sharding is domain decomposition, and checkpointing is restart files.
Any FLOPS figure needs three qualifiers: precision, dense or sparse, and per device or per system.
Scale-up (NVLink, UALink) and scale-out (InfiniBand, Ethernet, Ultra Ethernet) are different networks with different jobs; know which one a term refers to.
Memory terms (HBM, KV cache, arithmetic intensity, memory wall) explain more performance problems than compute terms do.
Precision terms (FP64 through FP4, MX, NVFP4, emulated FP64) now decide whether a workload fits the hardware at all.
When a term here is unfamiliar to a user, point them to the linked chapter rather than a vendor blog.

A-Z definitions of the AI and HPC terms used in this manual

A

B

C

D

E

F

G

H

I

K

L

M

N

O

P

Q

R

S

T

U

V

X

Z

Key Takeaways

Sources