Memory and Storage

In the 2019 edition of this manual, "storage" meant a Lustre or GPFS scratch file system, an NFS home server and a tape library, and "memory" meant choosing DIMMs so every channel was populated. Both topics have moved to the center of cluster design. On a modern GPU node the scarce things are bytes per second out of HBM, bytes of HBM per GPU, and the rate at which the file system can absorb a checkpoint from a few thousand GPUs at once. This chapter works through the memory hierarchy from the top (HBM on the accelerator package) to the bottom (object storage and archive), with the math you need to decide whether a model fits, whether a data pipeline will starve the GPUs, and how much storage bandwidth to buy. Accelerator details are in Accelerators, fabrics in Interconnects, and parallelism strategies in Training at Scale.

The Memory Wall

The phrase "memory wall" goes back to Wulf and McKee in 1995: if processors keep getting faster than DRAM, eventually every program waits on memory. Gholami and colleagues revisited the trend for AI hardware in 2024 and found that over the previous 20 years peak server FLOPS grew about 3.0x every two years, while DRAM bandwidth grew 1.6x and interconnect bandwidth 1.4x over the same interval. Over the whole period they measured roughly a 60,000x increase in peak compute against about 100x for DRAM bandwidth and 30x for interconnect bandwidth.[1] Their conclusion, which matches what most of us see on real clusters, is that memory rather than compute is now the primary bottleneck for many AI workloads, especially inference with decoder (GPT-style) models.[1]

Arithmetic Intensity in One Paragraph

Arithmetic intensity is FLOPs performed per byte moved from memory.[1] A GPU with P FLOP/s of compute and B bytes/s of memory bandwidth needs an intensity of at least P/B to stay compute-bound. Large dense matrix multiplies at big batch sizes clear that bar. Many other things do not: element-wise ops, normalization, attention over long contexts, sparse and graph workloads, stencil codes, and above all autoregressive decoding, where each new token requires reading every weight once while doing only a couple of FLOPs per weight. For the classic HPC side, the same logic is why STREAM and HPCG track memory bandwidth far more closely than HPL does.

A Decode Back-of-Envelope

Take a 70-billion-parameter dense model stored in BF16 (2 bytes per parameter): 140 GB of weights. At batch size 1, generating one token means streaming all 140 GB from HBM. On a GPU with 8 TB/s of HBM bandwidth (the AMD Instinct MI355X figure[17]), that takes at least 140 / 8,000 s = 17.5 ms per token, a hard ceiling of about 57 tokens/s per sequence no matter how many FLOPs the chip has (and 140 GB exceeds the capacity of many single GPUs anyway). Batching many sequences amortizes the weight reads, which is why serving systems batch aggressively, and why the KV cache (below) becomes the capacity problem. This is an upper-bound calculation, not a benchmark; see Inference at Scale for real serving behavior.

Field note: When a user says "my job only gets 30 percent GPU utilization," check memory before anything else. Nsight Compute or rocprof will tell you whether kernels are bandwidth-bound. A faster GPU generation with the same HBM bandwidth will not help a bandwidth-bound code much; more bandwidth per GPU will.

High Bandwidth Memory (HBM)

HBM stacks DRAM dies vertically with through-silicon vias and places the stack next to the GPU die on a silicon interposer or similar package, giving a very wide, short interface. A GPU's total memory bandwidth is essentially the per-stack bandwidth times the number of stacks, and its capacity is stack capacity times stack count. That is why HBM generation matters as much as the GPU architecture name.

Generations and Per-Stack Numbers

Note: HBM2E and HBM3E rows use memory-maker product figures, not a JEDEC press release. Vendor HBM4 parts run faster than the JEDEC baseline pin rate.

HBM4 Status as of September 2026

JEDEC published JESD270-4 (HBM4) in April 2025. It doubles the interface to 2048 bits and the channel count from 16 to 32, defines 8 Gb/s pins for up to 2 TB/s per stack, supports 4-, 8-, 12- and 16-high stacks of 24 Gb or 32 Gb dies (up to 64 GB per cube), adds Directed Refresh Management for row-hammer mitigation, and keeps controller compatibility with HBM3.[5] A revision, JESD270-4A (version 1.1), followed in December 2025.[6]

• Samsung announced mass production and commercial shipment of HBM4 on 12 February 2026: 11.7 Gb/s pins (capable of 13 Gb/s), up to 3.3 TB/s per stack, 24-36 GB in 12-high stacks, with 16-high stacks planned to reach 48 GB.[9] Samsung began sampling 12-high, 48 GB HBM4E at 14 Gb/s (scalable to 16 Gb/s, up to 3.6 TB/s per stack) on 29 May 2026; that part is sampling, not in volume.[10]
• SK hynix lists HBM4 at 11.0 Gb/s and higher (2.8 TB/s and higher) in 24-48 GB densities with 8-, 12- and 16-high stacks.[3]
• Micron lists HBM4 with a 2048-bit interface at more than 11.0 Gb/s and more than 2.8 TB/s per stack, a 36 GB 12-high part in production and 48 GB 16-high samples in 2026.[8]

The practical meaning: 2026-generation accelerators such as NVIDIA Rubin (288 GB HBM4 at 19.2 TB/s per GPU on the current Vera Rubin NVL72 page; NVIDIA's launch materials said 22 TB/s)[16][16] and AMD Instinct MI455X (432 GB HBM4, up to 23.3 TB/s)[18] roughly double or better the bandwidth of their HBM3E predecessors.

Accelerator Memory at a Glance

Will It Fit? GPU Memory vs Model Size

Most "CUDA out of memory" tickets can be answered with arithmetic before anyone opens a profiler. The four big consumers are weights, gradients and optimizer states (training only), activations (training only, mostly), and the KV cache (inference and long-context training).

Weights

Weights = parameters x bytes per parameter. FP32 is 4 bytes, BF16/FP16 2, FP8 1, and 4-bit formats half a byte (plus a small overhead for scale factors). The ZeRO paper gives the canonical example: the 1.5B-parameter GPT-2 needs only 3 GB for its weights in 16-bit precision.[21]

Training: the 16 Bytes per Parameter Rule

Mixed-precision training with Adam keeps an FP16/BF16 copy of the weights (2 bytes per parameter) and gradients (2 bytes), plus FP32 optimizer states: a master copy of the weights, momentum and variance (4 + 4 + 4 = 12 bytes). Total: 16 bytes per parameter before any activations.[21] That is why the same GPT-2 that has 3 GB of weights needs at least 24 GB of model state to train, and could not be trained on a single 32 GB GPU with plain data parallelism.[21] Activations are extra; ZeRO reports about 60 GB of activations for GPT-2 at sequence length 1K and batch 32, which activation checkpointing (recomputation) reduces substantially at a compute cost.[21]

Worked example (70B, full fine-tune or pretraining): 70 x 10^9 x 16 bytes = 1,120 GB of model state. Even on 288 GB GPUs that is at least four GPUs of pure state with nothing left for activations, so in practice you shard with ZeRO/FSDP across a node or more, and add tensor or pipeline parallelism as the model grows. ZeRO-style sharding divides model state across data-parallel ranks (the paper cites a 64x reduction across 64 GPUs).[21] ZeRO-Infinity goes further and offloads to CPU memory and NVMe, motivated by the observation that the largest dense models grew over 1000x in three years while GPU memory grew 5x (16 GB to 80 GB).[62]

Inference: the KV Cache

For each token in the context, every layer stores a key and a value vector. Per token: 2 x layers x KV heads x head dimension x bytes per element. The vLLM paper works the example for OPT-13B (hidden size 5120, 40 layers, FP16, no grouped-query attention): 800 KB per token, or up to 1.6 GB for one 2048-token request.[22] KV cache fragmentation was bad enough that paging it like virtual memory (PagedAttention) improved serving throughput 2-4x in their evaluation.[22]

Worked example (Llama 3 70B): the published architecture has 80 layers, model dimension 8192, 64 attention heads and 8 KV heads (grouped-query attention).[23] Head dimension is 8192 / 64 = 128. Per token in BF16: 2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes (320 KiB). At a 32,768-token context that is about 10.7 GB per sequence; 16 concurrent sequences need about 172 GB of KV cache on top of 140 GB of BF16 weights. The same arithmetic for Llama 3 405B (126 layers, 8 KV heads, dimension 16,384 with 128 heads)[23] gives 504 KiB per token.

# Example sizing helper: memory footprint of a dense transformer (illustrative)
GB = 1e9

def weights_gb(params, bytes_per_param):
    return params * bytes_per_param / GB

def train_states_gb(params, bytes_per_param=16):
    # Mixed-precision Adam: 2 (fp16 weights) + 2 (fp16 grads) + 12 (fp32 master, m, v)
    return params * bytes_per_param / GB

def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_elem=2):
    return 2 * layers * kv_heads * head_dim * bytes_per_elem

def kv_cache_gb(layers, kv_heads, head_dim, context, batch, bytes_per_elem=2):
    return kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_elem) * context * batch / GB

if __name__ == "__main__":
    p = 70e9  # Llama 3 70B: 80 layers, 8 KV heads, head_dim 8192/64 = 128
    print(f"BF16 weights: {weights_gb(p, 2):.0f} GB")                 # 140
    print(f"FP8 weights:  {weights_gb(p, 1):.0f} GB")                 # 70
    print(f"Adam model states: {train_states_gb(p):.0f} GB")          # 1120
    print(f"KV per token: {kv_bytes_per_token(80, 8, 128) / 1024:.0f} KiB")  # 320
    print(f"KV, 32K ctx x 16 seqs: {kv_cache_gb(80, 8, 128, 32768, 16):.0f} GB")  # 172

Field note: Put this arithmetic on your user-facing documentation page and in your consultation intake form. Half of the "which GPU should I request" questions we get are answered by "parameters x 2 for inference, parameters x 16 plus activations for training, plus KV cache for long contexts." It also stops people requesting eight GPUs for a model that fits on one.

Beyond HBM: Coherent Memory, Host DRAM and CXL

The PCIe-attached GPU treats host memory as a distant place reached by explicit copies. Several current platforms remove that boundary, and host DRAM and CXL add capacity tiers below HBM.

NVLink-C2C on Grace Hopper and Grace Blackwell

The GH200 Grace Hopper Superchip connects a Grace CPU and Hopper GPU with NVLink-C2C at 900 GB/s of coherent bandwidth, which NVIDIA describes as 7x PCIe Gen5, and the CPU and GPU share a single per-process page table so GPU threads can access all system-allocated memory whether it physically sits in CPU LPDDR5X or GPU HBM.[20] The GB200 superchip pairs one Grace CPU (up to 480 GB LPDDR5X at up to 512 GB/s) with two Blackwell GPUs (372 GB HBM3E at 16 TB/s combined) over NVLink-C2C.[14] A GB300 NVL72 rack lists 20 TB of HBM3E plus 17 TB of LPDDR5X, which NVIDIA sums as 37 TB of "fast memory."[15] The Vera Rubin superchip raises NVLink-C2C to 1.8 TB/s.[16]

What this buys you: large embedding tables, vector indexes, KV cache overflow, optimizer state offload and out-of-core scientific arrays can live in CPU memory and be touched directly by the GPU at far above PCIe speed. What it does not buy you: HBM bandwidth. LPDDR5X at hundreds of GB/s per socket is an order of magnitude below HBM, so hot data still has to be in HBM.

AMD Instinct MI300A

MI300A goes further: 24 Zen 4 CPU cores and 228 CDNA 3 compute units share 128 GB of HBM3 in a single address space with 5.3 TB/s peak bandwidth.[19] There is no separate host DRAM to copy from, which makes it attractive for HPC codes that are hard to port with explicit data movement. The trade-off is a fixed 128 GB per socket.[19] See Scientific Computing in the AI Era for porting strategy.

DDR5 and MRDIMM

Host memory still matters on GPU nodes: data loader workers decode and augment in host RAM, asynchronous checkpointing stages copies there, and CPU-only HPC partitions depend on it entirely. Current x86 server platforms have 12 DDR5 channels per socket. AMD EPYC 9005 supports 12 channels of DDR5-6400 and up to 6 TB per socket at two DIMMs per channel, plus CXL 2.0.[24] A DDR5 channel is 64 data bits (8 bytes), so the theoretical peak is 6,400 MT/s x 8 bytes = 51.2 GB/s per channel, or 614.4 GB/s per 12-channel socket; the CXL Consortium uses the round figure of 50 GB/s for DDR5-6400.[31] Sustained STREAM numbers will be lower.

MRDIMMs (multiplexed rank DIMMs) run two ranks in parallel behind a data buffer to double the effective transfer rate at the module interface. Intel Xeon 6 supports MRDIMMs at up to 8,800 MT/s, which Intel says delivers more than 37 percent more bandwidth than standard RDIMMs, alongside DDR5-6400 RDIMMs and up to 12 memory channels.[25] JEDEC published a second-generation MRDIMM data buffer standard (JESD82-552) in April 2026 and reported the MRDIMM Gen2 module standard nearing completion, with raw cards targeting 12,800 MT/s still in development.[26]

Field note: Always populate every memory channel on every socket, with identical DIMMs. A GPU node with half its channels empty can bottleneck data loading and asynchronous checkpoint staging long before the GPUs notice anything else wrong. For CPU partitions running memory-bound HPC codes, MRDIMMs are worth pricing against simply buying more nodes.

CXL Memory Expansion and Pooling

Compute Express Link runs cache-coherent memory semantics over the PCIe physical layer. The spec history, per the CXL Consortium:[31]

• CXL 1.0 / 1.1 (2019): direct-attached devices at 32 GT/s.
• CXL 2.0 (November 2020): switching for fan-out, memory pooling so capacity can be assigned on demand rather than overprovisioned per server, and persistent memory support.[27]
• CXL 3.0 (August 2022): 64 GT/s with no added latency over 2.0, multi-headed and fabric-attached devices, multi-level switching, enhanced pooling and coherent memory sharing, peer-to-peer.[28]
• CXL 3.1 (November 2023): port-based-routing fabric switches, Global Integrated Memory for host-to-host communication, and a TEE security protocol for confidential computing with CXL memory.[29]
• CXL 3.2 (December 2024) followed, and CXL 4.0 (November 2025) doubles bandwidth to 128 GT/s, adds bundled ports and native x2 links, enhances memory RAS, and stays backward compatible with 3.x, 2.0, 1.1 and 1.0.[30][31]

Status check: a specification release is not a product. What you can buy and deploy today on mainstream servers is direct-attached CXL memory expansion on CPUs that support CXL 2.0 (EPYC 9005 is one example[24]). Switched pooling and shared-memory fabrics are in demonstrations and early products; ask vendors for shipping part numbers and supported OS versions before planning around them. For AI nodes, CXL memory is a capacity tier with higher latency than local DDR5, useful for large in-memory datasets, vector databases and KV-cache offload, not a substitute for HBM.

# Inspect NUMA layout; CXL expanders commonly show up as a memory-only node
numactl --hardware

# Example: bind a memory-hungry preprocessing job's allocations to node 2
# (replace 2 with the node number reported on your system)
numactl --cpunodebind=0 --membind=2 python preprocess.py

Storage for AI: What the I/O Actually Looks Like

NVIDIA's DGX SuperPOD reference architecture puts it plainly: the key I/O operation in deep learning training is re-read. The dataset is read over and over across epochs, checkpoints can be terabytes, and checkpoint writes are typically synchronous and block training.[32] There are three distinct patterns to design for.

Data Loading Throughput

Training reads are mostly large, parallel and read-dominant, but the access pattern depends on the format. NVIDIA recommends planning for 4 GB/s of read per GPU for vision workloads with high-resolution images where datasets exceed 30 TB, while noting that LLM training usually needs less read bandwidth.[32] Its reference storage tiers for one GB200 scalable unit are 40 GB/s read and 20 GB/s write (standard) or 125 GB/s read and 62 GB/s write (enhanced), scaling to 160/80 and 500/250 GB/s for four units.[32] Caching the dataset in client RAM or on local NVMe after the first epoch can be an order of magnitude faster than going back to remote storage.[32]

Checkpoint Write Bursts

Checkpointing is a burst: every rank writes at once, the job waits (unless you checkpoint asynchronously), and then the file system is idle until the next one. Meta's Llama 3 paper describes a 240 PB Tectonic storage fabric on 7,500 SSD servers sustaining 2 TB/s with 7 TB/s peak, where "a major challenge is supporting the highly bursty checkpoint writes that saturate the storage fabric for short durations," with per-GPU checkpoints from 1 MB to 4 GB.[23] NVIDIA's guidance is to provision write bandwidth of at least half the read bandwidth for LLM and large-model use cases.[32]

The Small-File Problem

Datasets of millions of small JPEGs, text snippets or per-sample NumPy files turn a bandwidth problem into a metadata problem. Every open, stat and close hits the metadata servers, and a parallel file system that streams hundreds of GB/s on large files can be brought to its knees by a single job listing a directory of ten million files from 500 data loader workers. The IO500 benchmark measures exactly this with its mdtest-hard phase: 3,901-byte files created, stat-ed, read and deleted in one shared directory.[37] The fix is almost always to repack data into large shards (next sections), not to buy more metadata servers.

Field note: Set inode (file count) quotas on scratch as well as capacity quotas, and alert on them. A user with 40 million files in scratch is a support ticket waiting to happen for everyone else on the file system. Teach sharding in your onboarding workshop; it is the cheapest storage performance upgrade you will ever make.

Parallel File Systems and Object Storage

All the systems below share the core idea of separating metadata from data and striping file contents across many servers so that aggregate bandwidth scales with server count. They differ in licensing, hardware model, client, metadata design and operational effort.

The Main Options

Lustre Striping in Practice

Striping spreads a file over multiple OSTs. Wide stripes help large shared files and checkpoints; small files should stay on one OST. Progressive File Layouts (Lustre 2.10 and later) let one directory default grow the stripe count as a file grows.[40]

# Fixed layout: 4 MiB stripes over 2 OSTs (the example from the Lustre wiki)
lfs setstripe -S 4M -c 2 /mnt/testfs/project

# Example PFL for a checkpoint directory: small files on 1 OST,
# growing to 4 OSTs, then all OSTs (-c -1) past 1 GiB to end of file (-E -1)
lfs setstripe -E 64M -c 1 -E 1G -c 4 -E -1 -c -1 /mnt/testfs/project/checkpoints

# Check what a file actually got
lfs getstripe /mnt/testfs/project/checkpoints/step_1000/shard_0000.distcp

Object Storage (S3) Tiers

Object storage (S3 API, on-premises or cloud) is now a first-class tier for AI data: cheap per TB, effectively unlimited namespace, good aggregate throughput for large objects, and weak at small random POSIX I/O. MLPerf Storage v3.0 (September 2026) added an S3 access layer alongside POSIX for training, checkpointing and some vector database tests, and about one-sixth of submissions used it; the working group co-chair said they expect object storage to become a viable, possibly preferred, alternative to file systems.[35] A common pattern is object storage as the system of record for datasets and archived checkpoints, a flash parallel file system as the hot tier, and local NVMe as a cache. WEKA and others fold the object tier into one namespace.[44] See Cloud and Hybrid AI/HPC for cloud object and managed Lustre options.

GPUDirect Storage, Local NVMe and Burst Buffers

GPUDirect Storage (GDS)

GDS provides a direct DMA path between storage and GPU memory that avoids a bounce buffer through CPU memory, reducing CPU load and latency and relieving host bandwidth bottlenecks.[48] Applications use the cuFile API directly or through libraries built on it. GDS is supported on third-party systems including DDN EXAScaler, IBM Spectrum Scale, NetApp ONTAP, BeeGFS, WekaFS, VAST NFS and Dell, and on Amazon FSx for Lustre; recent releases added an NVMe P2PDMA mode using upstream Linux NVMe drivers (kernel 6.2 or later, or RHEL 9.4's 5.14 kernel on x86_64).[49] When the direct path is unavailable, cuFile falls back to a compatibility mode, so "it works" does not prove "it is using GDS." The gdscheck tool reports which modes are supported, and runtime settings live in /etc/cufile.json.[49][50]

# Verify the GDS install and which modes are available (path per NVIDIA docs)
/usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p

# Runtime configuration lives in /etc/cufile.json; logs go to cufile.log

GDS matters most when the GPU is the first and last to touch the data (GPU decode, scientific arrays loaded straight to device) and when CPU memory bandwidth is the bottleneck.[48] For a typical PyTorch pipeline doing JPEG decode and augmentation on CPU workers, the bottleneck is elsewhere.

Local NVMe Scratch and Burst Buffers

Every current GPU node ships with local NVMe, and NVIDIA recommends using it to cache or stage data.[32] Uses: stage a dataset once per job so every epoch reads locally; write checkpoints locally first and drain them to shared storage in the background; hold container images and compilation caches. MLPerf Storage's checkpoint benchmark even includes a "subset" mode representing node-local NVMe as the first checkpoint tier, with the caveat that it does not by itself provide fault tolerance.[33]

Burst buffers generalize this. LLNL's El Capitan uses "Rabbit" near-node storage: each Rabbit node has 16 SSDs and an AMD EPYC storage processor, two per rack, each directly connected to 16 compute nodes, orchestrated by Flux; a static XFS allocation of about 200 GB per node is mounted at /l/ssd on batch nodes.[51]

#!/bin/bash
#SBATCH --job-name=stage-and-train
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --time=24:00:00
# Example only: LOCAL path and shard layout are site/project specific.

LOCAL=/local/scratch/$SLURM_JOB_ID
SRC=/lustre/project/datasets/imagenet-wds

# One task per node creates the local dir and copies the shards
srun --ntasks=$SLURM_NNODES --ntasks-per-node=1 bash -c "
  mkdir -p $LOCAL && cp $SRC/shard-*.tar $LOCAL/
"

srun python train.py --data "$LOCAL/shard-{000000..001023}.tar"

# Clean up local scratch on every node
srun --ntasks=$SLURM_NNODES --ntasks-per-node=1 rm -rf "$LOCAL"

Field note: Make node-local scratch a scheduler-managed resource (per-job directory created in a prolog, wiped in an epilog) rather than a free-for-all /tmp. Otherwise the first job that fills the NVMe breaks the next five jobs on that node. See Scheduling and Orchestration.

Data Formats and Pipelines

Format choice decides whether your storage sees a few large sequential reads or millions of tiny random ones.

Formats for Machine Learning

• WebDataset: plain POSIX tar files where files belonging to one sample share a basename, with shards numbered for brace expansion (data-{000000..012345}.tar). It reads from local disk or any pipe, including object stores.[52]
• Parquet: open-source column-oriented format with compression and encoding schemes, the default for tabular and text corpora.[53]
• TFRecord: a sequence of binary records, usually serialized tf.train.Example protocol buffers.[54]
• Streaming datasets: MosaicML Streaming reads sharded datasets from remote object storage with local caching and resumable, deterministic shuffling.[58] Hugging Face datasets streams with streaming=True; its documentation example is the 45 TB English split of FineWeb.[59]

# WebDataset fluid interface (adapted from the project README)
import webdataset as wds

url = "/local/scratch/shards/train-{000000..001023}.tar"   # example path
dataset = (
    wds.WebDataset(url, shardshuffle=100)
    .shuffle(1000)            # in-memory sample shuffle buffer
    .decode("pil")
    .to_tuple("png", "json")
)

Formats for Science

• HDF5: self-describing hierarchical container with parallel I/O through MPI-IO; tune alignment to the file system block size and pass MPI-IO hints.[56]
• NetCDF-4: the climate and geoscience standard, stored on disk as HDF5.[57]
• Zarr: chunked, compressed N-dimensional typed arrays, stored as many chunk objects, which makes it a natural fit for object storage and cloud-native analysis (core specification version 3.1).[55]

For AI-for-science projects, the pattern that works is to keep the canonical data in HDF5, NetCDF or Zarr, and generate training shards (WebDataset, Parquet, or large NumPy/Zarr chunks aligned to the sample) as a versioned derived product rather than having thousands of data loader workers each open the canonical file.

Field note: Target shard sizes in the hundreds of MB to low GB. Much smaller and you are back to metadata storms; much larger and shuffling granularity and restart times suffer. Record the shard manifest and random seed with the checkpoint so a restarted job resumes the same data order.

Checkpointing Strategies and Sizes

Why and How Often

Synchronous training stalls when any one component fails. In Llama 3 pretraining on 16K GPUs, Meta saw 466 job interruptions in a 54-day snapshot, 419 unexpected, about 78 percent attributed to confirmed or suspected hardware issues, while still achieving more than 90 percent effective training time.[23] The MLPerf Storage working group built a failure model from that data and applied the rule of thumb of losing no more than 5 percent of progress: about 20 checkpoints between expected failures. That yields a checkpoint every 9.3 minutes at 16,000 accelerators (with about 28 seconds allowed per checkpoint to keep overhead under 5 percent) and every 1.5 minutes at 100,000 accelerators (4.4 seconds allowed).[33] A 64-GPU university cluster will fail far less often per job, but the same method applies: estimate job-level MTBF from your own incident data and set the interval from it.

How Big Is a Checkpoint?

A full training checkpoint is dominated by optimizer state, not weights. MLPerf Storage v2.0's checkpoint workload uses these sizes:[33]

Note: sizes and process counts from MLCommons[33]; the last column is simple division. Optimizer state is 90 GB of the 8B checkpoint and 13.2 TB of the 1T checkpoint.[33]

Worked example: writing the 912 GB 70B checkpoint takes about 46 seconds at the 20 GB/s "standard" single-SU write rate from NVIDIA's reference, or about 15 seconds at the 62 GB/s "enhanced" rate.[32][33] MLCommons notes that at 100,000 accelerators a 15 TB checkpoint every 1.5 minutes is more than 14 PB of writes per day, and writing it synchronously in under 5 seconds needs about 3.6 TB/s.[33] MLPerf Storage also reports checkpoint time by the slowest process, because a checkpoint is not done until every rank is done.[33]

Strategies That Work

• Sharded, parallel checkpoints. Each rank writes its own shard (PyTorch Distributed Checkpoint, DeepSpeed, Megatron) instead of gathering to rank 0. This spreads the burst across all OSTs or storage servers.
• Asynchronous save. PyTorch DCP's async_save copies state to CPU buffers and writes in the background. The cost is host memory (checkpoint size per rank times ranks per node) and the need to manage outstanding saves; PyTorch recommends at most one in flight. A pinned-memory staging buffer speeds the copy but stays allocated between checkpoints.[60]
• Tiered checkpoints. Write to local NVMe or peer memory frequently, drain to the parallel file system less often, and copy selected checkpoints to object storage or archive.
• Retention policy. Keep the last few checkpoints plus milestones. Five retained 70B checkpoints is 4.5 TB per run.

# Asynchronous DCP save, one outstanding request at a time
# (pattern from the PyTorch async checkpoint recipe)
import torch.distributed.checkpoint as dcp

checkpoint_future = None
for step in range(num_steps):
    train_step()
    if step % ckpt_every == 0:
        if checkpoint_future is not None:
            checkpoint_future.result()      # wait for the previous save
        state_dict = {"app": AppState(model, optimizer)}
        checkpoint_future = dcp.async_save(
            state_dict, checkpoint_id=f"/lustre/project/ckpt/step{step}"
        )

More on checkpoint/restart mechanics and elastic training is in Training at Scale.

Data Management, Lifecycle and Research Data Security

Lifecycle Tiers

Write the policy down and enforce it with tooling, not email. A workable tier model:

• Node-local NVMe: per-job, wiped at job end.
• Scratch (parallel file system): high performance, not backed up, purged after a fixed period of no access, inode and capacity quotas.
• Project (parallel file system or NAS): quota-managed per group, snapshots, for active datasets, code and selected checkpoints.
• Object / campus storage: system of record for datasets and model artifacts, versioned, lifecycle rules to cold tiers.
• Archive: tape or cold cloud tier for data that must be retained for publication, grant or regulatory reasons.

Track provenance: which dataset version, shard manifest, code commit and container produced each model. Funders increasingly require data management and sharing plans; the storage tiers above are where those plans become real.

Research Data Security

AI training concentrates sensitive data (clinical notes, genomic data, student records, export-controlled simulation output) into shards and checkpoints that are easy to copy and hard to audit. Models trained on sensitive data can themselves be sensitive artifacts. For federally funded work with Controlled Unclassified Information, NIST SP 800-171 Revision 3 is the governing control set for nonfederal systems.[61] Practical controls at the storage layer: separate file systems or filesets per security enclave; encryption at rest; POSIX ACLs and project groups rather than world-readable scratch; logging of access to regulated datasets; and making sure checkpoints, KV caches written to disk and derived shards inherit the classification of their source data. Governance and enclave operations are covered in Research Computing Operations.

Field note: The purge policy on scratch is also a security control. Regulated data copied to scratch "just for this run" and never deleted is a common audit finding. Pair purges with a clear, documented place where regulated data is allowed to live.

Benchmarks and Center Sizing

MLPerf Storage

MLPerf Storage emulates accelerators by replacing compute with measured "think time," so it generates realistic storage patterns without real GPUs.[34] The training workloads pass only if the emulated accelerators stay busy: minimum accelerator utilization is 90 percent for 3D U-Net (KiTS19, 140 MB samples) and ResNet-50 (ImageNet, 150 KB samples) and 70 percent for CosmoFlow (2 MB samples).[36] The headline result is therefore "how many accelerators can this storage keep fed," which is exactly the number you need for procurement. v2.0 (August 2025) added checkpointing and drew more than 200 results from 26 organizations, with systems supporting roughly twice as many accelerators as in v1.0.[34] v3.0 (September 2026) added KV cache and vector database tests and an S3 access layer, with 19 submitting organizations.[35]

IO500

IO500 is the HPC community's storage benchmark suite, combining IOR and mdtest phases: ior-easy (each process writes its own file at maximum throughput), ior-hard (interleaved 47,008-byte records to one shared file), mdtest-easy (zero-byte files in per-process directories), mdtest-hard (3,901-byte files in one shared directory) and a find phase, each run against a 300-second stonewall timer.[37] Separate lists cover production and research systems plus a 10-node challenge; Argonne's Aurora DAOS system took the top production spot at SC23.[38]

Use both as references, not purchase criteria by themselves. Ask vendors for results at the node count and client configuration you will actually run, run the benchmarks yourself during acceptance, and remember the mdtest-hard phase is the closest proxy for the small-file behavior your users will generate.

A Sizing Worksheet for a Center

The worksheet below is a method, not a quote. Replace the example values with your own measurements. The example is a hypothetical 64-GPU cluster (8 nodes x 8 GPUs) serving a mixed campus population.

Two sanity checks. First, the worst case in line B is rarely the average: if most of your GPU hours are LLM fine-tuning and inference, read demand is lower and checkpoint write bursts dominate, so weight line D more heavily. Second, compare against MLPerf Storage results: find submissions that sustain your target accelerator count at the required utilization, and see what hardware they needed.[34][36] Power and floor space for storage are real line items too; see Power, Cooling and Facilities, and instrument all of this per Observability, Performance and Efficiency so next year's worksheet uses measured numbers.

Key Takeaways

• Peak compute has grown much faster than memory and interconnect bandwidth; many AI workloads, especially decoding, are bound by HBM bandwidth and capacity, not FLOPS.
• HBM4 is standardized (JESD270-4, 2025) and shipping in 2026 accelerators, with vendor parts at 2.8-3.3 TB/s per stack, above the JEDEC 2 TB/s baseline; HBM4E is sampling.
• Fit models with arithmetic first: 2 bytes per parameter for BF16 inference, about 16 bytes per parameter plus activations for mixed-precision Adam training, plus KV cache that grows with context x concurrent sequences.
• Coherent CPU-GPU memory (NVLink-C2C, MI300A) and CXL expand capacity, not HBM bandwidth; treat them as a warm tier for offload, embeddings, indexes and KV overflow.
• Design storage for three patterns: repeated dataset reads, synchronous checkpoint write bursts, and small-file metadata storms. Size writes at half of reads or better for LLM work.
• Shard data into large files (WebDataset, Parquet, TFRecord, chunked Zarr/HDF5), stage to local NVMe, and enforce inode quotas; it is cheaper than more hardware.
• Checkpoints are mostly optimizer state (about 13-15 bytes per parameter in MLPerf's workload); use sharded, asynchronous, tiered checkpointing and a retention policy.
• Use MLPerf Storage (accelerators kept busy) and IO500 (bandwidth plus metadata) as procurement references, and verify at your own scale during acceptance.

Sources

1. Gholami et al., AI and Memory Wall (arXiv 2403.14123, IEEE Micro, 2024)
2. JEDEC Updates Groundbreaking High Bandwidth Memory (HBM) Standard, JESD235B (JEDEC, 2018)
3. HBM product lineup: HBM2E, HBM3, HBM3E, HBM4 (SK hynix, accessed 2026)
4. JEDEC Publishes HBM3 Update to High Bandwidth Memory (HBM) Standard, JESD238 (JEDEC, 2022)
5. JEDEC and Industry Leaders Collaborate to Release JESD270-4 HBM4 Standard (JEDEC, 2025)
6. High Bandwidth Memory (HBM4) DRAM, JESD270-4A (JEDEC, 2025)
7. HBM3E (Micron, accessed 2026)
8. HBM4 (Micron, accessed 2026)
9. Samsung Ships Industry-First Commercial HBM4 (Samsung Newsroom, 2026)
10. Samsung Electronics Begins Shipment of Industry-First HBM4E Samples (Samsung Newsroom, 2026)
11. NVIDIA A100 Tensor Core GPU Datasheet (NVIDIA)
12. NVIDIA H100 Tensor Core GPU (NVIDIA, accessed 2026)
13. NVIDIA H200 GPU (NVIDIA, accessed 2026)
14. NVIDIA GB200 NVL72 (NVIDIA, accessed 2026)
15. NVIDIA GB300 NVL72 (NVIDIA, accessed 2026)
16. NVIDIA Vera Rubin NVL72 (NVIDIA, accessed 2026)
17. AMD Instinct MI355X GPU Brochure (AMD, 2025)
18. AMD Instinct MI400 Series GPUs (AMD, accessed 2026)
19. AMD Instinct MI300A APU Data Sheet (AMD, 2025)
20. NVIDIA GH200 Grace Hopper Superchip (NVIDIA, accessed 2026)
21. Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv 1910.02054, 2020)
22. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP, 2023)
23. Llama Team, The Llama 3 Herd of Models (arXiv 2407.21783, Meta, 2024)
24. AMD EPYC 9005 Series Server CPUs Datasheet (AMD)
25. The Intel Xeon 6 Processor Family Product Brief (Intel)
26. JEDEC Advances DDR5 MRDIMM Ecosystem with New Memory Interface Logic (JEDEC, 2026)
27. CXL Consortium Releases Compute Express Link 2.0 Specification (CXL Consortium via Business Wire, 2020)
28. CXL Consortium Releases Compute Express Link 3.0 Specification (CXL Consortium, 2022)
29. Introducing Compute Express Link 3.1 (CXL Consortium white paper, 2023)
30. CXL Consortium Releases the Compute Express Link 4.0 Specification (CXL Consortium, 2025)
31. Introducing the CXL 4.0 Specification, webinar slides (CXL Consortium, 2025)
32. DGX SuperPOD GB200 Reference Architecture: Storage Architecture (NVIDIA)
33. Announcing the MLPerf Storage v2.0 Checkpointing Workload (MLCommons, 2025)
34. New MLPerf Storage v2.0 Benchmark Results (MLCommons, 2025)
35. MLCommons Releases New MLPerf Storage v3.0 Benchmark Results (MLCommons, 2026)
36. MLPerf Storage Submission Guidelines (MLCommons, GitHub)
37. About the IO500 (IO500 Foundation)
38. IO500 News (IO500 Foundation)
39. Lustre Architecture for Admins (Lustre Wiki)
40. Configuring Lustre File Striping (Lustre Wiki)
41. BeeGFS Architecture Overview (BeeGFS Documentation)
42. DAOS Architecture (DAOS v2.4 Documentation)
43. DAOS on Aurora (Argonne Leadership Computing Facility User Guides)
44. WEKA Data Platform Introduction (WEKA Documentation 4.3)
45. What is Disaggregated Shared Everything (DASE)? (VAST Data)
46. EXAScaler Lustre File System (DDN)
47. IBM Storage Scale (IBM)
48. GPUDirect Storage Overview Guide (NVIDIA)
49. NVIDIA GPUDirect Storage Release Notes (NVIDIA)
50. GPUDirect Storage Installation and Troubleshooting Guide (NVIDIA)
51. Using El Capitan Systems: File Systems and Rabbits (LLNL)
52. WebDataset (GitHub)
53. Apache Parquet Overview (Apache Software Foundation)
54. TFRecord and tf.train.Example (TensorFlow)
55. Zarr Core Specification Version 3.1 (Zarr)
56. Parallel HDF5 (The HDF Group)
57. NetCDF-4 File Format, NetCDF User's Guide (NSF Unidata)
58. StreamingDataset (MosaicML Streaming Documentation)
59. Stream (Hugging Face Datasets Documentation)
60. Asynchronous Saving with Distributed Checkpoint (DCP) (PyTorch, 2026)
61. NIST SP 800-171 Rev. 3, Protecting Controlled Unclassified Information in Nonfederal Systems (NIST)
62. Rajbhandari et al., ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (arXiv 2104.07857, 2021)