Accelerators: GPUs, TPUs and Beyond

The 2019 edition of this manual was written for CPU clusters. In 2026 the accelerator decides almost everything else: rack power, cooling, network topology, storage bandwidth, the scheduler setup, and what your users can actually do. This chapter covers how these chips work, which ones are shipping and which are only announced, and how a research computing center should choose among them. Every hard number here comes from a vendor datasheet, vendor documentation or a peer-reviewed source listed at the end. Where a vendor has not published a figure, this chapter says so and does not guess. Treat the tables as a starting point and re-check the datasheet before you sign a purchase order. Specs for announced products change.

GPU Architecture Fundamentals for Practitioners

You do not need to write CUDA kernels to run a GPU cluster well, but you do need the mental model. Without it you cannot explain to a user why their job reaches 8% utilization, or why one card with twice the FLOPS runs their code no faster.

SMs, CUs and the Execution Model

An NVIDIA GPU is built around an array of Streaming Multiprocessors (SMs).[26] AMD calls the same building block a Compute Unit (CU). Each SM or CU holds scalar and vector ALUs, a register file, a block of fast on-chip memory, and (on data center parts) matrix units. A Blackwell Ultra GPU has 160 SMs across two dies, and each SM has four fifth-generation Tensor Cores, 640 in total.[8] An AMD MI300X is built from chiplets. Each Accelerator Complex Die (XCD) carries 38 CUs,[28] for 304 CUs on the full device.[18]

Threads are scheduled in fixed-size groups that execute the same instruction together. In CUDA this group is a warp of 32 threads, and when threads in a warp take different branches, the inactive lanes are masked off while the others execute.[26] AMD's documentation calls the group a warp (HIP terminology) or wavefront (ISA terminology). It is 64 threads on CDNA data center architectures and 32 threads on RDNA consumer and workstation architectures.[25] This matters when porting code: kernels that hard-code a warp size of 32 can break or slow down on Instinct hardware.

The key idea is latency hiding. When a warp stalls on a memory load from HBM, the scheduler switches to another resident warp that is ready to run. Because every resident thread's registers stay on the chip, this switch costs almost nothing.[25] A GPU therefore needs many more threads in flight than it has ALUs. That is why small problems, tiny batch sizes and serial code run poorly on it.

Tensor Cores and Matrix Cores

NVIDIA introduced Tensor Cores with Volta. Instead of working one element at a time, they perform a small matrix multiply-accumulate in a single instruction.[8] AMD's equivalent is the Matrix Core (1,216 on MI300X[18]). Intel's Gaudi uses Matrix Multiplication Engines (MMEs),[35] and Google's TPUs use Matrix Multiply Units (MXUs).[39] Almost all of the headline FLOPS on a modern accelerator come from these units, and mostly at low precision (BF16, FP8, FP4). A code that does not reduce to dense matrix multiplication gets the much smaller vector throughput instead. The gap is large. On MI355X, peak FP16 matrix throughput is 2.5 PFLOPS dense, while plain FP16 vector throughput is 157.3 TFLOPS.[22]

The On-Die Memory Hierarchy

From fastest to slowest, the levels are registers, then per-SM shared memory and L1 (software-managed scratchpad plus cache), then a large shared L2 or last-level cache, then HBM, then host memory over PCIe or a coherent CPU link. Some reference points:

• On Hopper (compute capability 9.0), the unified L1/shared memory is 256 KB per SM, and one thread block can address up to 227 KB of shared memory.[56]
• Blackwell-generation Tensor Cores are paired with 256 KB of Tensor Memory (TMEM) per SM, placed next to the Tensor Cores.[8]
• MI300X and MI355X each have a 256 MB last-level Infinity Cache.[18][22]
• Gaudi 3 has 96 MB of on-die SRAM.[35] Ironwood TPUs expose a software-managed on-chip vector memory (VMEM) as a scratchpad for custom kernels.[38]

The Memory and Storage chapter covers HBM in depth.

Why GPUs Win on Throughput

A CPU spends its transistors on making one thread fast: big caches, branch prediction, out-of-order execution. A GPU spends them on ALUs and memory bandwidth, and relies on massive parallelism to hide latency. For work that is data-parallel and arithmetic-dense (dense linear algebra, stencils, FFTs, particle methods, and above all neural network training and inference), that trade wins by a wide margin in both throughput and energy per operation. It loses on branchy, serial, latency-bound code. Your job as a facilitator is often to work out which kind of code a user actually has.

Field note: Before you blame the hardware for a slow GPU job, check three things: whether the work is actually on the GPU (not silently falling back to the CPU), whether the batch or problem size is large enough to fill the device, and whether the job spends its time in data loading. nvidia-smi or amd-smi metric utilization near zero with a busy CPU usually means the input pipeline is the bottleneck, not the accelerator. The Observability chapter covers this properly.

# Quick device inventory from inside a job (works on CUDA and ROCm builds of PyTorch)
import torch

if not torch.cuda.is_available():
    raise SystemExit("No GPU visible - check your Slurm --gres/--gpus request")

for i in range(torch.cuda.device_count()):
    p = torch.cuda.get_device_properties(i)
    print(f"GPU {i}: {p.name}, {p.total_memory / 2**30:.0f} GiB, "
          f"{p.multi_processor_count} SMs/CUs")

NVIDIA Data Center GPUs: Volta to Rubin

NVIDIA now releases a new data center architecture about once a year, with mid-cycle "Ultra" refreshes. What follows is the lineage most research centers will have on the floor or in a procurement right now.

Volta, Ampere and Hopper

Volta V100 introduced Tensor Cores: 640 of them, delivering 125 TFLOPS of tensor performance on the SXM2 part, with 32 or 16 GB of HBM2 at 900 GB/s and 300 GB/s of NVLink.[1] Many campuses still run V100 nodes. They remain useful for teaching and for FP64 codes that fit in 32 GB. Before you plan around them, check that the CUDA and framework versions your users need still support Volta, and plan their retirement.

Ampere A100 brought TF32 and BF16, structured sparsity, FP64 Tensor Cores (19.5 TFLOPS) and Multi-Instance GPU (MIG). The 80 GB SXM part runs at 2,039 GB/s and 400 W.[2]

Hopper H100 added FP8 and the Transformer Engine, fourth-generation NVLink at 900 GB/s, and PCIe Gen5. The SXM part delivers 34 TFLOPS FP64 vector and 67 TFLOPS FP64 Tensor Core, with 80 GB at 3.35 TB/s and up to 700 W.[3] H200 keeps the same compute and raises memory to 141 GB of HBM3E at 4.8 TB/s.[4] For memory-bound inference and for HPC codes limited by bandwidth, that change alone is a real upgrade.

Blackwell: B200 and GB200 NVL72

Blackwell GPUs are two reticle-limited dies joined by a 10 TB/s chip-to-chip link and presented to software as one GPU. The package has 208 billion transistors on TSMC 4NP.[10] Blackwell adds FP4 (NVIDIA's NVFP4 format) and fifth-generation NVLink at 1.8 TB/s per GPU.[8] In the HGX B200 baseboard each GPU has 180 GB of HBM3E at up to 8 TB/s.[9] A DGX B200 (8 GPUs, 10U) draws about 14.3 kW maximum.[11]

The GB200 NVL72 is the rack-scale version. It holds 36 Grace CPUs and 72 Blackwell GPUs in one NVLink domain with 130 TB/s of aggregate NVLink bandwidth, 13.4 TB of HBM3E and 17 TB of LPDDR5X CPU memory.[6] The basic building block is the GB200 superchip: one Grace CPU and two Blackwell GPUs.[6]

Blackwell Ultra: B300 and GB300 NVL72

Blackwell Ultra keeps the two-die, 208-billion-transistor design. NVIDIA rebalanced it toward inference: 1.5x the dense NVFP4 throughput of Blackwell (15 vs 10 PFLOPS), 288 GB of HBM3E at 8 TB/s, doubled special-function throughput for attention softmax, and a power ceiling of up to 1,400 W.[8] The GB300 NVL72 rack has 20 TB of HBM and 37 TB of total fast memory.[7]

The part that matters for HPC is where those transistors came from. Blackwell Ultra gives up most of its FP64 and INT8 throughput. NVIDIA's own HGX spec table lists FP64/FP64 Tensor Core at 296 TFLOPS for an 8-GPU HGX B200 and 10 TFLOPS for an 8-GPU HGX B300. At rack scale it is 2,880 TFLOPS for GB200 NVL72 and 100 TFLOPS for GB300 NVL72.[5][6][7] INT8 drops in the same way (see the table below). A B300 is a very good inference and low-precision training part. It is the wrong choice for a double-precision simulation code.

Vera Rubin and the Announced Roadmap

NVIDIA's Rubin GPU and Vera CPU (88 custom Arm-compatible Olympus cores) make up the Vera Rubin platform.[5] NVIDIA's Vera Rubin NVL72 page says the platform is "now in full production" and that Taiwanese server makers are shipping Vera Rubin systems.[13] NVIDIA's GTC 2026 coverage says Microsoft Azure was the first hyperscaler to power up Vera Rubin NVL72, with a global rollout over the following months.[15] So Rubin is shipping in 2026, but in the early ramp and mostly to hyperscalers. Most campus buyers will see it later.

NVIDIA's preliminary per-GPU Rubin figures:[5]

• 288 GB of HBM4 at 22 TB/s, and 3.6 TB/s of sixth-generation NVLink.
• 35 PFLOPS dense NVFP4 for training, and 50 PFLOPS NVFP4 for inference (NVIDIA labels this figure sparse).
• FP64: 33 TFLOPS native, plus a separate 200 TFLOPS "FP64 DGEMM" figure that NVIDIA footnotes as peak performance using Tensor Core-based emulation.

The rack uses the third-generation MGX NVL72 design, which NVIDIA presents as a continuation of the GB200/GB300 rack.[13]

Further out, and still roadmap only:

• NVIDIA has said its next major architecture after Rubin is Feynman.[15]
• Trade press coverage of GTC 2026 reports that Rubin Ultra (four compute chiplets, 1 TB of HBM4E) and the 144-GPU Kyber rack are planned for the second half of 2027.[14]

Plan facilities against these dates with caution. The racks keep getting denser, and the Facilities chapter explains why that matters more than the FLOPS.

NVIDIA Comparison Tables

The tables are split so each one fits on a laptop screen. Table 1 covers per-GPU memory, bandwidth and power.

Field note: NVIDIA has published two Rubin figures: 22 TB/s HBM4 bandwidth and 3.6 TB/s NVLink per GPU at launch (HGX Rubin NVL8 spec and Rubin platform announcement), and 19.2 TB/s HBM4 and 3 TB/s NVLink per GPU on the current Vera Rubin NVL72 product page. Treat Rubin numbers as preliminary and check the vendor page for your configuration.[5][13]

Note: B200/B300 max power is the "Max power (TGP)" row in NVIDIA's Blackwell Ultra comparison table.[8] NVIDIA quotes NVLink bandwidth as bidirectional. "Not listed" means the figure is not in the cited NVIDIA spec table.

Table 2 covers per-GPU AI throughput. Figures are dense unless marked.

Table 3 is the table HPC buyers need: 8-GPU HGX baseboards, with the precisions that matter for science.

Note: NVIDIA publishes the Blackwell tensor figures as sparse and states that dense is one-half of the sparse figure. The dense values above apply that stated rule. NVIDIA publishes Rubin figures as dense.[5]

Table 4 compares NVL72 racks.

AMD Instinct and ROCm

AMD Instinct GPUs power two of the TOP500's exascale-class machines. In the June 2026 list, El Capitan (MI300A) is No. 2 at 1.809 EFLOP/s with 60.94 GFLOPS/W energy efficiency, and Frontier (MI250X[60]) is No. 3.[54] AMD has kept strong FP64 in every generation, and that is the main technical reason HPC centers look at it.

MI250X Through MI355X

• MI250X (CDNA2, launched November 2021) was the Frontier GPU: 128 GB HBM2e at 3.2 TB/s, 47.9 TFLOPS FP64 vector and 95.7 TFLOPS FP64 matrix, OAM form factor.[17] Each MI250X has two Graphics Compute Dies, and software sees each die as a separate GPU with 64 GB of HBM2E, so a 4-card Frontier node appears as 8 GPUs.[60]
• MI300X (CDNA3, December 2023): 192 GB HBM3 at 5.3 TB/s, 81.7 TFLOPS FP64 vector and 163.4 TFLOPS FP64 matrix, 750 W.[18] For its generation it offered a lot of memory per GPU, which made it popular for LLM inference.
• MI300A is an APU: 24 Zen 4 CPU cores and 228 CDNA3 CUs sharing 128 GB of unified HBM3 at 5.3 TB/s, in a socket (SH5) rather than on a card.[19] CPU and GPU share one memory pool, so there are no host-device copies. Porting and memory management get much simpler, which is why El Capitan uses it.
• MI325X is a memory upgrade of MI300X: 256 GB HBM3E at 6 TB/s, at 1,000 W.[20]
• MI350X / MI355X (CDNA4, June 2025): 288 GB HBM3E at 8 TB/s, with MXFP4 and MXFP6 support. MI355X is the higher-clocked, 1,400 W variant.[21][22]

Note: AMD publishes both dense and "with structured sparsity" figures. Dense is shown. The MI455X figures are AMD's "peak" values, and AMD does not label them dense or sparse on the product page.[23] The MI325X FP16 value is the 1,307.4 TFLOPS figure from AMD's footnote.[20]

One detail matters for HPC buyers. On CDNA3 (MI300X, MI325X), FP64 matrix throughput is double the vector rate. On CDNA4 (MI350X, MI355X), FP64 matrix equals FP64 vector and is lower than on MI300X.[18][22] AMD also moved silicon toward low precision in this generation, just less dramatically than NVIDIA did with B300.

The Announced MI400 Series

AMD's MI400 series splits into two products:[23]

• MI455X is for frontier AI: 432 GB of HBM4 at 23.3 TB/s per GPU, built into AMD's Helios rack. Helios holds 72 MI455X GPUs, 31 TB of HBM4 and up to 2.9 exaFLOPS of MXFP4, in Meta's double-wide Open Rack Wide (ORW) form factor, using open standards including UALink and UEC. AMD says Helios is a reference design for OEM and ODM partners, not a product AMD sells, with volume deployments expected in the second half of 2026.
• MI430X is aimed squarely at science: 288 TFLOPS of hardware FP64 and 432 GB of HBM4. AMD expects it in 2027 and names ORNL's Discovery system and the European Alice Recoque and Herder systems as planned deployments.

AMD positions MI430X as the FP64 alternative to NVIDIA's emulation-heavy approach. Take that seriously, and remember it is not shipping yet.

ROCm in 2026

ROCm is AMD's open-source software stack. It includes the HIP programming model (close enough to CUDA that most kernels port mechanically), math libraries, RCCL for collectives, and profilers. The current release as of this writing is ROCm Core SDK 10.0.0, dated 2026-08-26. Since 7.14, ROCm uses the TheRock build and release system.[24] AMD says it provides Day-0 support for PyTorch, JAX, vLLM and Triton on the MI400 series.[23]

In practice, mainstream PyTorch training and vLLM serving are well supported on MI300-class hardware. The gaps show up in the long tail: custom CUDA extensions, niche research codes, and tools that assume nvcc. The Systems Software chapter covers driver and container management for both stacks.

Field note: Before you commit to AMD, take your top 20 user applications by core-hours and check each one for HIP or ROCm support, a container that runs, and a named person who has run it on Instinct. For most centers the major MD, climate, CFD and PyTorch codes pass. The one lab with a hand-written CUDA code and a graduated student is the one that will email the provost. Budget facilitation time for porting, or keep a smaller NVIDIA partition.

Intel: Max Series, Gaudi 3 and What Comes Next

Data Center GPU Max (Ponte Vecchio) is the GPU in Argonne's Aurora:[34]

• 10,624 nodes, 63,744 GPUs, six Max 1550 GPUs and two Xeon CPU Max processors per node.
• Each GPU has two stacks and 128 Xe cores. Each Xe core has 8 vector engines and 8 matrix engines.
• Each GPU has a 384 MiB last-level cache, and each node has 768 GiB of GPU HBM2e.

Aurora holds No. 4 on the June 2026 TOP500 at 1.012 EFLOP/s.[54] Its software path is oneAPI and SYCL. Intel has not positioned Max as a broad product line for new buyers. Treat it as a leadership-facility platform rather than something to procure for a campus.

Gaudi 3 is Intel's AI accelerator, a different architecture from the Max GPUs:[35]

• Two compute dies with 8 MMEs and 64 Tensor Processor Cores in total.
• 1,678 TFLOPS of BF16 and FP8 matrix compute.
• 128 GB of HBM2e at 3.7 TB/s, and 96 MB of on-die SRAM.
• 24 integrated 200 GbE RDMA ports, which make scale-out standard Ethernet.
• Up to 900 W in the OCP OAM 2.0 module.

Intel positions it for AI training and inference, and its white paper lists no FP64 throughput. Trade press reports that Gaudi fell short of Intel's sales expectations. An Intel executive said publicly in January 2026 that the company "did not meet the needs of the frontier AI training market."[37]

Intel's current direction is a yearly GPU cadence, and it is focused on inference. Crescent Island, announced October 2025, is an Xe3P data center GPU with 160 GB of LPDDR5X, designed for air-cooled enterprise servers, with customer sampling expected in the second half of 2026.[36] Intel cancelled Falcon Shores in 2025. Trade press reports that the follow-on rack-scale product, earlier called Jaguar Shores, is now described as the "Shores product line."[37] For a 2026 procurement, Intel accelerators are a niche choice, mainly Gaudi 3 where the price is right and the models are supported.

Google TPUs and Cloud Access

Google's Tensor Processing Units are systolic-array ASICs that you access only through Google Cloud, via Compute Engine or GKE.[38] They are built as pods: chips linked by a dedicated inter-chip interconnect (ICI) in a 2D or 3D torus, with the data center network between slices.

A few notes on each generation:

• v5e and v6e are the cost-efficient training and serving parts, with slices of 1 to 256 chips. v6e supports a single-host 8-chip VM (ct6e-standard-8t) optimized for inference.[40]
• v5p is the large-scale training part. The largest single job is 6,144 chips.[41]
• Ironwood (TPU7x) has two TensorCores and four SparseCores per chip, and Google's documentation describes it as the latest TPU available on Google Cloud.[38]

In April 2026 Google announced an eighth generation split into two chips:[42]

• TPU 8t for pre-training: 216 GB HBM, 12.6 PFLOPS FP4, 9,600 chips per superpod.
• TPU 8i for serving and reasoning: 288 GB HBM, 384 MB of on-chip SRAM, 10.1 PFLOPS FP4.

Both use Arm Axion host CPUs. Trade press reported at the announcement that both chips would be generally available later in 2026.[43] Check current regional availability before planning around them.

On software, TPUs are programmed through XLA: JAX first, with PyTorch/XLA and Keras also supported. Google says native PyTorch support for TPUs, including eager mode, is in preview.[42] Google's TPU documentation lists BF16, FP8 and INT8 throughput and no FP64 figures, so treat TPUs as AI accelerators, not double-precision simulation engines. For research centers they make sense when a group already works in JAX, when cloud credits or a partnership cover the cost, or when you need a large pod for a short time. The Cloud and Hybrid chapter covers the logistics.

Other Accelerators: Wafers, Dataflow and Hyperscaler Silicon

Cerebras

Cerebras builds one processor per silicon wafer. The WSE-3 (announced March 2024, TSMC 5nm) has 4 trillion transistors, 900,000 AI cores, 44 GB of on-chip SRAM and 125 PFLOPS of peak AI performance, and powers the CS-3 system.[44] Cerebras now markets a WSE-3 Turbo rated at 250 PFLOPS, and a CS-4 rack system that combines three WSE-3T processors for inference.[45] Because the whole model's working set sits in on-wafer SRAM, token generation is very fast. Argonne has worked with Cerebras for years.[44] For most campuses, Cerebras is something you use through a cloud API or a national facility allocation, not hardware you rack yourself.

Groq and NVIDIA Groq 3 LPX

Groq built SRAM-centric Language Processing Units (LPUs) for low-latency inference. The landscape changed in 2026. NVIDIA now ships the NVIDIA Groq 3 LPX rack as an extension of the Vera Rubin platform, with Groq and LPU marks "used under license from Groq, Inc." NVIDIA announced it was in full production at Hot Chips 2026, with Nebius as the first cloud adopter and Groq's own inference cloud among early adopters.[16] The design pairs Rubin GPUs, which hold large contexts in HBM, with LPUs, which generate tokens from SRAM.[46] For research computing this is a cloud-consumption story, not a procurement one.

SambaNova

SambaNova's SN40L Reconfigurable Dataflow Unit combines streaming dataflow with a three-tier memory system so that one system can hold and switch between many expert models (the paper's Samba-CoE deployment has 150 experts and a trillion total parameters). The design is described in a MICRO 2024 paper.[48] In February 2026 Intel announced a multiyear strategic collaboration with SambaNova.[37]

AWS Trainium and Inferentia

AWS builds its own chips and offers them only as EC2 instances, programmed through the Neuron SDK:

• Trn2 instances have 16 Trainium2 chips and 20.8 FP8 PFLOPS. Trn2 UltraServers link 64 chips over NeuronLink.[49]
• Trn3 UltraServers use Trainium3, AWS's first 3nm AI chip. They scale to 144 chips with up to 362 MXFP8 PFLOPS, 20.7 TB of HBM3e and 706 TB/s of aggregate memory bandwidth, over a NeuronSwitch all-to-all fabric.[50]
• Inf2 instances carry up to 12 Inferentia2 chips with 32 GB of HBM each, for serving.[51]

AWS claims 30-40% better price-performance for Trn2 than for its GPU-based P5e and P5en instances.[49] That is a vendor claim, so test it on your own model before relying on it.

Grace and CPU-GPU Superchips

NVIDIA's Grace CPU has 72 Arm Neoverse V2 cores with LPDDR5X memory at up to 500 GB/s. The Grace CPU Superchip pairs two of them.[52] In GH200, a Grace CPU connects to a Hopper GPU over NVLink-C2C at 900 GB/s, with a single per-process page table, so CPU and GPU threads can access all system-allocated memory.[53] GB200 and GB300 use the same idea with Blackwell GPUs.

This coherent-memory design is the practical answer to "my problem does not fit in HBM". JUPITER, Europe's first exascale system, uses Grace Hopper.[54] AMD's MI300A reaches the same goal a different way, by putting CPU and GPU in one package with shared HBM.[19] Superchips mean Arm (for NVIDIA) and a new node design, so check that your user codes and your provisioning stack handle aarch64 before you buy.

The FP64 Question for Scientific Computing

For twenty years, "GPU for HPC" meant strong FP64. That assumption no longer holds. AI revenue pays for the silicon, and AI does not need double precision. The Convergence chapter covers precision more generally. Here is the hardware picture.

FP64 Throughput Trends

Three different paths are visible in the table:

• NVIDIA peaked at Hopper and Blackwell, cut FP64 almost entirely in Blackwell Ultra, and in Rubin offers modest native FP64 plus emulation.
• AMD's CDNA4 parts trade some FP64 for low precision, and the announced MI430X goes the other way with a dedicated FP64 part.
• TPUs and the other AI ASICs covered in this chapter do not publish FP64 figures at all.

Emulation and the Ozaki Scheme

The Ozaki scheme splits FP64 matrices into several low-precision slices, multiplies the slices on fast integer or low-precision tensor units, and recombines the results so the final product matches FP64 accuracy. Ozaki Scheme II uses the Chinese Remainder Theorem to control the number of multiplications. On INT8 Tensor Cores it reported 56.6 to 80.2 TFLOPS of emulated FP64 on a GH200, above that chip's native FP64 rate.[55]

NVIDIA shipped this in cuBLAS starting with CUDA 13.0 Update 2, as FP64 matmul emulation on Blackwell INT8 Tensor Cores (plus a BF16x9 FP32 emulation). An automatic dynamic precision framework chooses emulation parameters that match or beat native FP64 accuracy, and falls back to native FP64 when emulation would not help.[57] NVIDIA reports a 1.5x end-to-end speedup for Quantum ESPRESSO with default settings on an RTX PRO 6000 Blackwell Server Edition, and nearly 3x with manually reduced precision.[57]

What this means in practice:

• Emulation speeds up matrix multiplication (GEMM and routines built on it). It does nothing for stencils, sparse solvers, FFT-heavy codes (research work on this exists[55]) or anything else that does not reduce to dense GEMM. Many important simulation codes are limited by memory bandwidth rather than GEMM throughput. For them, HBM bandwidth matters far more than FP64 peak.
• Emulation needs fast low-precision integer units. Blackwell Ultra cut INT8 along with FP64 (Table 3), so B300 is a poor emulation target as well as a poor native FP64 target.
• NVIDIA's Rubin DGEMM figure (200 TFLOPS per GPU) is an emulated peak, not a native one.[5] Compare like with like when a vendor puts native and emulated numbers side by side.
• Validate numerics for each application. The accuracy guarantee covers the GEMM, and your code's error behavior is your own responsibility.

Field note: Do not write "FP64 TFLOPS" as a single line item in a procurement. Ask vendors for results on your actual HPC codes (and HPL, HPCG and STREAM as sanity checks), say whether emulation is allowed, and ask for the native figure separately. A B300 system and a B200 system look similar on an AI spec sheet and differ by about 30x on native FP64.

GPU Partitioning: MIG, MPS, Time Slicing and AMD Modes

A 180 GB, 1,000 W GPU is too much for a student's Jupyter notebook. Partitioning lets you share it. The Scheduling chapter covers how Slurm and Kubernetes expose these partitions. This section covers the hardware mechanisms.

NVIDIA Multi-Instance GPU (MIG)

MIG splits a supported GPU into isolated instances, each with dedicated compute, memory and cache. It works on GPUs from Ampere onward (compute capability 8.0 and higher), and on DGX, HGX and DGX Station systems built on them.[31] H100 supports up to 7 instances of 10 GB each, and H200 up to 7 instances of 18 GB each.[3][4] Instances are created with nvidia-smi.[30]

# Enable MIG on GPU 0 (requires root; drain the node first)
sudo nvidia-smi -i 0 -mig 1

# List the GPU instance profiles this GPU supports
nvidia-smi mig -lgip

# Example: create two instances by profile name, with default compute instances (-C)
# (profile names differ by GPU model - use the names that -lgip shows you)
sudo nvidia-smi mig -cgi 3g.20gb,3g.20gb -C

# Tear down: destroy compute instances, then GPU instances
sudo nvidia-smi mig -dci && sudo nvidia-smi mig -dgi

MPS and Time Slicing

MPS (Multi-Process Service) lets several CUDA processes share one GPU at the same time through a server process. It is typically used for MPI ranks that each underuse the GPU. Client processes get isolated address spaces, but error containment is limited. A fatal fault in one client is reported to every client sharing that GPU. Monitoring tools attribute all client activity to the MPS server.[32] Use MPS for cooperating processes from one user or job, not for mutually untrusted tenants. You can also run MPS inside MIG instances.[30]

Time slicing oversubscribes a GPU by handing out "replicas" that take turns. NVIDIA's GPU Operator documentation is explicit that, unlike MIG, time slicing provides no memory or fault isolation between replicas. It is also the only sharing option on GPUs that do not support MIG.[33]

# Kubernetes GPU Operator time-slicing config (from NVIDIA's documented format)
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4   # example value: 4 pods share each physical GPU

AMD Compute and Memory Partitioning

MI300-series parts separate compute partitioning (SPX, DPX, CPX) from memory partitioning (NPS1, NPS2, NPS4). AMD's partitioning index lists SPX + NPS1, DPX + NPS2 and CPX + NPS4 for MI300X, and only SPX + NPS1 for MI300A.[27] An MI300X has eight XCDs.[59] In CPX mode each XCD becomes its own logical GPU, so an 8x MI300X server reports 64 GPUs to amd-smi.[59] Partitions are configured with amd-smi:[29]

# Show current compute and memory partition modes
sudo amd-smi partition --current
sudo amd-smi partition --memory
sudo amd-smi partition --accelerator   # lists valid accelerator partition values

# Example: set memory partition mode on GPU 0 (use values reported above)
sudo amd-smi set -g 0 --memory-partition NPS4

# Example from AMD's documentation: all GPUs to CPX compute partitioning
sudo amd-smi set --gpu all --compute-partition CPX

Field note: Reconfiguring partitions needs an idle GPU and root access, so do not let users change modes themselves. Choose a small number of fixed node "shapes" (for example, a few nodes in 7-way MIG for classes and notebooks, the rest whole-GPU), label them in the scheduler, and resize the pools each semester based on queue data.

Form Factors: PCIe, SXM, OAM and Rack-Scale

The same GPU name can ship in very different packages, and the package decides what you can do with it.

• PCIe cards fit standard servers and air cooling, and usually cost less per GPU. They run at lower power and have weaker GPU-to-GPU links. The H100 NVL (PCIe) runs at 350-400 W with 600 GB/s NVLink over a bridge, against the SXM part's 700 W and 900 GB/s.[3] Use PCIe for inference, single-GPU jobs, visualization and mixed-use nodes.
• SXM modules on an HGX baseboard are NVIDIA's 4- or 8-GPU building block. All GPUs connect through NVLink Switch (1.8 TB/s per GPU and 14.4 TB/s aggregate on HGX B200/B300).[5] OEMs build their servers around HGX, and DGX is NVIDIA's own branded system (DGX B300: 10U, about 14 kW).[12] Use SXM for multi-GPU training and for tightly coupled HPC.
• OAM (OCP Accelerator Module) is the open equivalent of SXM, used by AMD Instinct and Intel Gaudi 3 on a universal baseboard.[18][35] MI355X OAMs have seven Infinity Fabric links for scale-up.[22]
• Rack-scale systems (GB200/GB300 NVL72, Vera Rubin NVL72, AMD Helios) make the whole rack one scale-up domain. They are liquid cooled and sold, delivered and serviced as a rack.[6][23] The benefit is a 72-GPU NVLink domain. The cost is facility work most campus data centers have not done yet.

The Interconnects chapter covers the scale-up and scale-out fabrics, and Facilities covers power and liquid cooling.

Choosing Accelerators for a Research Center

Start With Workload Fit

Pull a year of scheduler accounting and group the GPU hours:

• AI training at scale (multi-node, weeks-long): SXM/OAM nodes with a full-bandwidth InfiniBand or RoCE fabric. See Training at Scale.
• Inference and fine-tuning: memory capacity and bandwidth per GPU matter most. Large-HBM parts (H200, B300, MI325X/MI355X) or PCIe cards for smaller models. See Inference at Scale.
• Double-precision simulation: native FP64 and HBM bandwidth. Today that means H100/H200, B200, MI300A/MI300X-class, or MI430X when it ships. See Scientific Computing.
• Teaching, notebooks, small models: MIG slices of a large GPU, or workstation-class cards.
• Bursty frontier-scale jobs: rent them. Cloud TPU pods, Trainium or GPU capacity cost less than an idle on-prem rack.

Weigh the Software Ecosystem Honestly

CUDA's advantage is breadth: every framework, nearly every research code, the most examples, and the most people who know how to debug it. ROCm is production-grade for the mainstream and needs more work in the long tail. XLA (TPU) and Neuron (Trainium) are strong for supported model families and narrow outside them. The cost of a less common stack is staff and student time. Either budget for it or accept a mixed fleet.

Total Cost and Power

Purchase price is one part of the cost, and increasingly not the largest. Per-GPU power went from 300 W (V100 SXM2) to up to 1,400 W (B300 and MI355X).[1][8][22] A single 8-GPU DGX draws about 14 kW.[11][12] Over a five-year life, a TCO model needs to include:

• Electricity and cooling.
• Facility upgrades (liquid cooling, busway, floor loading).
• Network ports: one high-speed NIC per GPU is now standard. DGX B300 has eight 800 Gb/s ConnectX-8 ports.[12]
• Software subscriptions where required, and staff.

Measure cost per useful result (per training run, per simulated nanosecond, per million tokens), not per peak TFLOP.

Procurement Tips

• Benchmark your own codes. Require vendors to run your application suite, and state precision, sparsity and emulation rules up front. Vendor peak numbers mix sparse and dense and native and emulated figures, as the notes under this chapter's tables show.
• Buy memory, not just FLOPS. For LLM work, HBM capacity decides which models fit on one node.
• Ask about delivery and lifecycle. Get lead times, how long the vendor will support the part, driver and CUDA/ROCm support windows, and RMA turnaround in writing.
• Plan the facility first. A rack-scale system your data center cannot power or cool is a very expensive pallet.
• Stay a generation behind when it makes sense. H200 and MI300X/MI325X-class systems are mature, well supported, and often the best value for mixed academic workloads.
• Keep a cloud option for peaks and for trying new silicon (TPU, Trainium, the latest NVIDIA and AMD parts) before you commit capital. See Cloud and Hybrid.

Field note: The most common mistake I see is a center buying the highest-spec AI GPU because it tops the leaderboard, when half the queue is FP64 simulation and the other half is notebooks that need 20 GB. Match the silicon to the queue you actually have. A split fleet of a few FP64-capable SXM nodes, some large-memory inference nodes and a MIG pool usually serves a campus better than one uniform expensive partition.

Workstation GPUs for Research

Workstation-class cards are useful on campus: for prototyping, visualization, small-model fine-tuning, and as low-cost cluster nodes. The NVIDIA RTX PRO 6000 Blackwell Workstation Edition has 96 GB of GDDR7 with ECC at 1,792 GB/s, 125 TFLOPS of single precision, 4,000 "AI TOPS" (a theoretical FP4 figure with sparsity) and a 600 W maximum, in a dual-slot PCIe Gen 5 card.[58] NVIDIA says the Max-Q variant supports MIG with up to four instances.[58] There is also a Server Edition for data center chassis.[57] 96 GB on one card lets a lab fine-tune or serve mid-sized models locally.

Know the limits before users treat these as cheap data center GPUs:

• GDDR7, not HBM. 1.8 TB/s against 8 TB/s on B200/B300.[58][8] Memory-bound training and HPC codes will notice.
• Weak FP64. NVIDIA does not headline FP64 for RTX PRO. Its own FP64-emulation blog uses the RTX PRO 6000 Server Edition as a showcase,[57] which tells you native double precision is not the card's strength.
• No NVLink Switch scale-up. The product page lists PCIe Gen 5 as the interconnect.[58] Multi-GPU training across these cards goes over PCIe and the network, and scales poorly for large models.
• Power and airflow. 600 W per card makes dense multi-card deskside builds a cooling problem, and office circuits a risk.

Use them for what they do well, and send users to the cluster when they hit these limits.

Key Takeaways

• GPUs win by running thousands of threads in parallel and doing matrix math on Tensor or Matrix Cores. Almost all headline FLOPS are low precision on those units, and codes that do not map to them see much less.
• NVIDIA's shipping lineup in 2026 runs from H100/H200 through B200 and B300 (Blackwell Ultra), with Vera Rubin in early production. Blackwell Ultra cut FP64 to 10 TFLOPS per 8-GPU HGX board (against 296 on HGX B200), so do not buy it for double-precision science.
• AMD Instinct (MI300A/MI300X through MI355X) offers large HBM capacity and strong FP64, and runs top exascale systems. The announced MI430X is a dedicated FP64 part for 2027. ROCm is solid for mainstream workloads, and the long tail needs porting time.
• Google TPUs (v5e through Ironwood, with TPU 8t/8i announced) and AWS Trainium are strong, cloud-only AI options. Neither targets FP64 simulation.
• FP64 emulation (Ozaki scheme in cuBLAS) can speed up GEMM-heavy codes on AI-oriented GPUs, but it does not help bandwidth-bound or non-GEMM codes. Always compare native with native when vendors quote emulated figures.
• Share big GPUs with MIG (isolated), MPS (cooperative) or time slicing (no isolation), or AMD's SPX/CPX and NPS modes. Offer a few fixed node shapes rather than letting users reconfigure.
• The form factor (PCIe, SXM/OAM, rack-scale NVL72 or Helios) decides power, cooling and scale-up capability. Plan the facility before the purchase.
• Choose from your own accounting data and your own benchmarks. A mixed fleet matched to the actual queue usually beats one uniform high-end partition.

Sources

1. NVIDIA V100 Tensor Core GPU Datasheet (NVIDIA, 2019)
2. NVIDIA A100 Tensor Core GPU product specifications (NVIDIA)
3. NVIDIA H100 Tensor Core GPU product specifications (NVIDIA)
4. NVIDIA H200 GPU product specifications (NVIDIA)
5. NVIDIA HGX Platform specifications: HGX B200, HGX B300, HGX Rubin NVL8 (NVIDIA, 2026)
6. NVIDIA GB200 NVL72 specifications (NVIDIA)
7. NVIDIA GB300 NVL72 specifications (NVIDIA)
8. Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era (NVIDIA Technical Blog, 2025)
9. NVIDIA HGX AI Factory Enterprise Reference Architecture: Components (NVIDIA)
10. NVIDIA Blackwell Architecture (NVIDIA)
11. NVIDIA DGX B200 specifications (NVIDIA)
12. NVIDIA DGX B300 specifications (NVIDIA)
13. NVIDIA Vera Rubin NVL72 (NVIDIA, 2026)
14. Nvidia updates data center product roadmap following LPU launch at GTC 2026 (DatacenterDynamics, 2026)
15. NVIDIA GTC 2026: Live Updates on What's Next in AI (NVIDIA Blog, 2026)
16. NVIDIA Groq 3 LPX Now in Full Production (NVIDIA Newsroom, 2026)
17. AMD Instinct MI250X Accelerators specifications (AMD)
18. AMD Instinct MI300X Accelerators specifications (AMD)
19. AMD Instinct MI300A Accelerators specifications (AMD)
20. AMD Instinct MI325X Accelerators and specification footnotes (AMD)
21. AMD Instinct MI350X GPUs specifications (AMD)
22. AMD Instinct MI355X GPUs specifications (AMD)
23. AMD Instinct MI400 Series GPUs: MI455X, MI430X and Helios (AMD, 2026)
24. ROCm Core SDK 10.0.0 release notes (AMD, 2026)
25. Introduction to the HIP programming model (AMD ROCm documentation)
26. CUDA Programming Guide: Programming Model (NVIDIA)
27. AMD GPU/APU Partitioning (AMD Instinct documentation)
28. AMD Instinct MI300X GPU Partitioning Overview (AMD Instinct documentation)
29. AMD SMI CLI tool documentation (AMD ROCm documentation)
30. NVIDIA MIG User Guide: Getting Started with MIG (NVIDIA)
31. NVIDIA MIG User Guide: Supported GPUs (NVIDIA)
32. NVIDIA Multi-Process Service: When to Use MPS and limitations (NVIDIA)
33. Time-Slicing GPUs in Kubernetes (NVIDIA GPU Operator documentation)
34. Aurora Machine Overview (Argonne Leadership Computing Facility User Guides)
35. Intel Gaudi 3 AI Accelerator White Paper (Intel)
36. Intel to Expand AI Accelerator Portfolio with New GPU (Intel Newsroom, 2025)
37. How Intel's AI Chip Strategy Is Coming Into Shape After Years Of Struggles (CRN, 2026)
38. TPU7x (Ironwood) documentation (Google Cloud)
39. TPU v5e documentation (Google Cloud)
40. TPU v6e (Trillium) documentation (Google Cloud, 2026)
41. TPU v5p documentation (Google Cloud)
42. Inside the eighth-generation TPU: An architecture deep dive (Google Cloud Blog, 2026)
43. Google unveils eighth-generation TPUs, two dedicated training and inference chips (DatacenterDynamics, 2026)
44. Cerebras Systems Unveils World's Fastest AI Chip with 4 Trillion Transistors (Cerebras, 2024)
45. Cerebras Wafer-Scale Engine 3 Turbo and CS-4 (Cerebras)
46. NVIDIA Vera Rubin Platform (NVIDIA, 2026)
47. NVIDIA DGX Vera Rubin NVL72 quick specs (NVIDIA, 2026)
48. Prabhakar et al., SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts (arXiv / MICRO 2024)
49. Amazon EC2 Trn2 instances and UltraServers (AWS)
50. Amazon EC2 Trn3 UltraServers (AWS)
51. Amazon EC2 Inf2 instances (AWS)
52. NVIDIA Grace CPU Superchip (NVIDIA)
53. NVIDIA GH200 Grace Hopper Superchip (NVIDIA)
54. TOP500 June 2026 list highlights (TOP500, 2026)
55. Ozaki et al., Ozaki Scheme II: A GEMM-oriented emulation of floating-point matrix multiplication using an integer modular technique (arXiv / IJHPCA, 2026)
56. CUDA C++ Programming Guide: compute capability 9.0 shared memory (NVIDIA)
57. Unlocking Tensor Core Performance with Floating Point Emulation in cuBLAS (NVIDIA Technical Blog, 2025)
58. NVIDIA RTX PRO 6000 Blackwell Workstation Edition (NVIDIA)
59. Deep dive into the MI300 compute and memory partition modes (AMD ROCm Blogs)
60. Frontier User Guide (Oak Ridge Leadership Computing Facility)