Inference at Scale

Training gets the headlines, but inference is where the GPU-hours go once a model is useful. Every chat session, agent loop and batch job that labels a million abstracts is inference, running every day the model is in service. Research computing groups that used to schedule batch jobs and walk away now also run long-lived, latency-sensitive services. This chapter covers how LLM inference works on the hardware, the metrics that matter, the KV cache and why it dominates memory planning, the serving engines worth knowing in 2026, quantization, multi-GPU layouts, sizing, RAG infrastructure, campus gateways, energy, and benchmarking. It ends with a working vLLM launch and a Slurm batch-inference job you can adapt.

Why Inference Dominates, and Why Decode Is Memory Bound

Training is a one-time cost per model; inference recurs and scales with users and output length. Google reported that across 2019-2021, about three fifths of its machine learning energy went to inference and two fifths to training[1]. Since then, reasoning models and agents have pushed the ratio further toward inference, because they generate many more tokens per request: the ML.ENERGY benchmark found that its Problem Solving (reasoning on) task produced on average about 10x more output tokens than its Text Conversation task (mean 6,988 vs. 717)[2].

Two Phases: Prefill and Decode

A decoder-only LLM serves a request in two phases[3][4]:

• Prefill. The whole prompt is processed in one forward pass. All prompt tokens are known, so the work is large matrix-matrix multiplies that keep the tensor cores busy. Prefill is compute bound. It produces the first output token and fills the KV cache for the prompt.
• Decode. Output tokens are generated one at a time. Each step processes one new token per sequence, but it has to read every model weight and the sequence's entire KV cache from HBM to do it. Decode is memory-bandwidth bound at small batch sizes.

A rough roofline makes the point. A dense transformer does roughly 2 FLOPs per parameter per token. In BF16 each parameter is 2 bytes, so a single-sequence decode step does about 1 FLOP per byte of weights read. An H100 SXM is listed at 3.35 TB/s of HBM bandwidth and 1,979 TFLOPS of BF16 tensor throughput, the latter quoted with sparsity[5]. Dividing the two gives about 590 FLOPs per byte at the quoted sparse peak; the dense rate is lower, but still far above the roughly 1 FLOP per byte that single-sequence decode needs. At batch size 1, the tensor cores sit mostly idle waiting on memory.

So every decode step must stream the weights at least once. For a 70.6 GB FP8 copy of a 70B model on one H200 (4.8 TB/s[6]), 70.6 GB / 4.8 TB/s = 14.7 ms per step, so no more than about 68 tokens per second for a single sequence, before counting KV cache reads. The only ways past that floor are fewer bytes (quantization), more bandwidth (newer GPUs, tensor parallelism) or more work per byte read (batching, speculative decoding). Most of this chapter is one of those three.

Field note: When a user asks "why is my 8-GPU node only at 30% utilization while serving?", the answer is usually that decode is waiting on HBM, not that something is broken. GPU "utilization" in nvidia-smi measures whether a kernel is running, not whether the tensor cores are saturated. Look at tokens per second and memory bandwidth counters instead (see Observability, Performance and Efficiency).

Metrics That Matter: TTFT, ITL, Throughput and Goodput

Inference has two audiences with different needs: a person watching tokens stream, and an operator paying for GPUs. The standard metrics map onto the two phases:

Throughput and latency pull against each other. Bigger batches raise tokens per second per GPU but make each step slower. Goodput, introduced by the DistServe paper, resolves this by counting only the requests that meet both a TTFT and a TPOT target[7]. It is the right number for capacity planning. vLLM's benchmark accepts SLOs directly, for example --goodput ttft:2000 tpot:50 (milliseconds)[8].

Set targets per use case: chat cares about TTFT and a TPOT faster than reading speed, a coding agent making chained calls cares about end-to-end latency, and an overnight batch job has no latency SLO at all. Write the SLOs down; they decide batch sizes, parallelism and whether you need disaggregation.

The KV Cache and Memory Management

The Number That Sizes Everything

During attention, each token's key and value vectors for every layer are needed by every later token. Recomputing them each step would make decode quadratic, so engines keep them in HBM: the KV cache. Its size per token is:

KV bytes per token = 2 (K and V) x num_layers x num_kv_heads x head_dim x bytes_per_element
KV bytes per sequence = KV bytes per token x sequence_length (prompt + generated)

Worked Example: Llama 3.3 70B

The published config for Llama 3.3 70B (shown here from Red Hat's FP8 checkpoint, which keeps the original architecture) lists 80 hidden layers, 64 attention heads, 8 key-value heads and a head dimension of 128[9]. With a BF16 cache (2 bytes per element):

2 x 80 layers x 8 kv_heads x 128 head_dim x 2 bytes = 327,680 bytes = 320 KiB per token

  8,192-token sequence  : 320 KiB x   8,192 =  2.5 GiB
 32,768-token sequence  : 320 KiB x  32,768 = 10.0 GiB
131,072-token sequence  : 320 KiB x 131,072 = 40.0 GiB

One full-length 128K-token conversation needs 40 GiB, half an 80 GB H100 on its own; an FP8 cache halves every number. This is why the KV cache, not the weights, usually limits concurrency, and why long-context and agentic workloads cost so much more than short chat.

GQA, MQA and MLA

Architecture choices change the num_kv_heads term, and that is the biggest lever on KV size:

• Multi-head attention (MHA) gives every query head its own K and V. If Llama 3.3 70B used MHA with 64 KV heads, the 128K sequence above would need 320 GiB.
• Multi-query attention (MQA) shares one K/V head across all query heads, cutting KV size and memory bandwidth during incremental decoding, with some quality cost[10]. At one KV head the same sequence would be 5 GiB.
• Grouped-query attention (GQA) uses an intermediate number of KV heads; its authors report quality close to MHA at speed comparable to MQA[11]. Llama 3.3 70B's 8 KV heads for 64 query heads is GQA, and it is why the real number is 40 GiB instead of 320.
• Multi-head latent attention (MLA), introduced with DeepSeek-V2, compresses keys and values into a shared latent vector. DeepSeek reported a 93.3% smaller KV cache than DeepSeek 67B and 5.76x higher maximum generation throughput[12]. The per-token formula above does not apply directly to MLA models; read the model's config and the engine's startup log instead.

Field note: Do not guess KV capacity; let the engine tell you. At startup vLLM logs a line like "GPU KV cache size: N tokens" and "Maximum concurrency for M tokens per request: Xx"[13]. If maximum concurrency at your chosen --max-model-len is in the single digits, lower the context limit, quantize the cache, or add GPUs before users find out for you.

Continuous Batching, PagedAttention and Prefix Caching

Continuous Batching

Static batching runs a batch until its longest request finishes, so short requests sit idle. The Orca system (OSDI 2022) introduced iteration-level scheduling: the scheduler makes decisions at every decode step, so finished sequences leave and new ones join the running batch immediately. Orca reported a 36.9x throughput improvement over NVIDIA FasterTransformer at the same latency on GPT-3 175B[14]. Every serious engine now does this; NVIDIA calls it in-flight batching[15].

PagedAttention

Earlier systems reserved a contiguous KV region sized for each request's maximum length, which wasted most of it. The vLLM paper (SOSP 2023) measured that only 20.4% to 38.2% of KV cache memory held actual token state in existing systems[16]. PagedAttention borrows virtual memory from operating systems: the KV cache is split into fixed-size blocks allocated on demand and tracked by a block table, so blocks need not be contiguous. That removes external fragmentation, limits internal fragmentation to the last block, and lets requests share blocks. The authors reported 2-4x throughput over FasterTransformer and Orca at the same latency[16].

Chunked Prefill

A long prompt arriving mid-stream can stall every decoding user while it prefills. Sarathi-Serve (OSDI 2024) splits prefills into chunks and mixes them with ongoing decodes in "stall-free" batches, reporting 2.6x higher serving capacity than vLLM for Mistral-7B on one A100 under tail-latency constraints[4]. Chunked prefill is now a standard scheduler feature in vLLM (--enable-chunked-prefill, with --max-num-batched-tokens as the per-step token budget)[17].

Prefix Caching

Many requests share a prefix: a system prompt, the same RAG documents, a growing agent history. Prefix caching reuses those KV blocks instead of recomputing prefill. SGLang's RadixAttention organizes cached prefixes as a radix tree and reported up to 6.4x higher throughput on workloads such as agents, few-shot prompts and multi-turn chat[18]. In vLLM, automatic prefix caching is on by default[19]. For agent workloads it is the most effective single optimization, which is why "KV-aware" routers send a conversation back to the same replica.

Serving Engines in 2026

For LLMs on a research cluster, start with vLLM or SGLang; use TensorRT-LLM when you are all-NVIDIA and want the last increment of performance; add Dynamo or Ray Serve when one engine instance is no longer enough; use llama.cpp or Ollama on workstations.

vLLM

vLLM is the default for most research groups: broad model coverage, an OpenAI-compatible server (vllm serve), an offline Python API[32], and tensor, pipeline, data and expert parallelism. It moves fast: the v0.29.0 release alone had 594 commits from 277 contributors and made a new model runner the default[20]. Pin a version per service and re-test before upgrading, because flags and defaults change between minor releases.

SGLang

SGLang grew out of the same Berkeley/LMSYS ecosystem, with RadixAttention prefix sharing and fast constrained (structured) decoding as its signature features[18]. The project states it runs on more than 400,000 GPUs in production[23]. It is a strong choice for agentic and multi-turn workloads with heavy prefix reuse.

TensorRT-LLM, Triton and Dynamo

NVIDIA's stack has three layers that are easy to confuse. TensorRT-LLM is the engine: optimized kernels, quantization (FP8, FP4) and a PyTorch-native LLM API[15]. Triton Inference Server is a general model server for any framework, with dynamic batching, ensembles and the KServe v2 protocol; it remains the right tool for non-LLM models such as vision, recommendation or tabular models[24]. Dynamo is the datacenter-scale orchestration layer that sits above vLLM, SGLang or TensorRT-LLM and adds disaggregated prefill/decode, KV-aware routing, multi-tier KV caching (GPU to CPU to SSD to remote storage) and an SLA-driven autoscaler[25]. Its v1.5.0 release (September 2026) deprecates the KV Block Manager (KVBM) component, with removal targeted for v1.6.0, while extending KV indexing to Mooncake and disk tiers[26]; check current docs before building on a specific KV-tiering component. Dynamo's own README is candid: if you are running a single model on a single GPU, the engine alone is probably enough[25].

Hugging Face TGI

Hugging Face has put Text Generation Inference in maintenance mode, accepting only minor fixes, and recommends vLLM, SGLang, llama.cpp or MLX going forward; the GitHub repository is now archived[28][29]. If you have TGI deployments, plan a migration; both vLLM and SGLang speak the same OpenAI-style API most clients use.

Small Scale: llama.cpp and Ollama

For a single workstation, a laptop, or CPU-only nodes, llama.cpp runs GGUF-format models with partial GPU offload (-ngl / --n-gpu-layers) and its llama-server exposes OpenAI-compatible chat, responses and embeddings routes[30]. Ollama wraps a similar experience with model management and also offers OpenAI-compatible /v1/chat/completions[31]. They are excellent for teaching, prototyping and single-user work, not for dozens of concurrent users on a shared node.

Ray Serve

Ray Serve LLM builds multi-node, multi-model deployments on Ray, with autoscaling, prefix-aware routing, multi-LoRA serving, prefill-decode disaggregation and vLLM or SGLang as the engine[27]. Its sibling, Ray Data LLM, is useful for offline batch inference over datasets larger than cluster memory[32].

Advanced Serving: Disaggregation, KV Offload and Speculation

Disaggregated Prefill/Decode and KV Cache Offload

Colocated prefill and decode interfere: a burst of long prompts inflates everyone's inter-token latency. Disaggregated serving runs them in separate pools; the prefill worker computes the KV cache and transfers it to a decode worker, which streams the response. DistServe reported up to 7.4x more requests or 12.6x tighter SLOs within latency constraints compared with colocated systems[7]. Moonshot AI's Mooncake, the serving platform for Kimi, separates prefill and decode clusters and pools spare CPU DRAM and SSD across the GPU cluster as a distributed KV cache; under real workloads it let Kimi handle 75% more requests[33].

Disaggregation is not free: the KV cache must move over NVLink or RDMA (see Interconnects). Dynamo's documentation states it plainly: for small models, short prompts, low concurrency or clusters without a fast KV-transfer fabric, an aggregated deployment is simpler and often faster[34]. vLLM labels its disaggregated prefill feature experimental and notes that it does not improve throughput; its benefit is separate tuning of TTFT and ITL and control of tail ITL[35].

KV Cache Offload and Tiering

KV cache can also be tiered: when HBM fills, blocks move to host DRAM, NVMe or remote storage and come back on a prefix hit, far cheaper than recomputing a long prefill. In vLLM, --kv-offloading-size sets a CPU offload buffer in GiB with a native or lmcache backend, and the --kv-transfer-config JSON selects connectors such as NixlConnector, MooncakeConnector, LMCacheConnectorV1 and OffloadingConnector[19][35]. Dynamo supports storage-tier KV offload including S3 and Azure blob[25], though its component layout is changing quickly[26]. See Memory and Storage.

Speculative Decoding

Speculative decoding gets more than one token per pass over the weights. A cheap drafter proposes several tokens; the large model verifies them in one compute-dense pass and accepts the longest correct prefix. With the right acceptance rule the output distribution is identical to the large model alone; Leviathan et al. reported 2-3x acceleration on T5-XXL with identical outputs[36].

Drafters include a small model from the same family, heads trained on the target's hidden states (EAGLE and EAGLE-3[37][38]), built-in multi-token prediction (MTP) layers, or model-free n-gram and suffix matching. vLLM configures all of these through one JSON flag, --speculative-config, with keys such as method, model and num_speculative_tokens[39].

Speculation spends compute to save bandwidth, so it helps most at low to medium load; vLLM's documentation frames it as a way to reduce inter-token latency for medium-to-low QPS, memory-bound workloads[39]. MLPerf Inference v6.1 now permits speculative decoding in the interactive scenario for some benchmarks, a sign it has become standard practice[40].

Field note: Measure acceptance rate on your traffic before enabling speculation on a shared service; recent vLLM releases report per-request acceptance statistics[20].

Quantization for Inference

Quantization cuts bytes per parameter, shrinking memory and raising the decode ceiling. Two styles: weight-only (for example W4A16: 4-bit weights, 16-bit activations), which helps bandwidth-bound decode most, and weight-and-activation (W8A8 in INT8 or FP8, or FP4), which also speeds up compute-bound prefill if the hardware has matching tensor-core formats.

INT8 and FP8

FP8 is the safe default on Hopper, Blackwell and MI300-class hardware[43][42]. vLLM's documentation describes FP8 W8A8 as a 2x reduction in model memory with up to 1.6x higher throughput and minimal accuracy impact; a simple round-to-nearest scheme with per-channel weights and dynamic per-token activations needs no calibration data[42]. Published FP8 checkpoints make the accuracy check easy to read: Red Hat's FP8-dynamic Llama 3.3 70B reports average recovery of 99.8% to 100.6% of the BF16 scores on its evaluation suites[46]. The KV cache can also be stored in FP8 (--kv-cache-dtype fp8) independently of the weights[19].

INT4: AWQ and GPTQ

GPTQ quantizes weights one layer at a time using approximate second-order information, reaching 3-4 bits per weight with small accuracy loss and quantizing 175B-parameter models in about four GPU hours[47]. AWQ observes that protecting about 1% of salient weight channels, identified from activation statistics, greatly reduces quantization error, and it does so without backpropagation[48]. Both yield weight-only 4-bit checkpoints that run on Ampere and even Turing GPUs, which lack FP8 tensor cores.

FP4 and NVFP4

Blackwell adds 4-bit floating point in hardware. NVFP4 stores E2M1 values with one FP8 (E4M3) scale per 16-value block plus a per-tensor FP32 scale, which works out to 4.5 bits per value and about 3.5x less memory than FP16 and 1.8x less than FP8[43]. MXFP4 uses a power-of-two scale per 32 values. NVIDIA reports 1% or less accuracy degradation versus FP8 for DeepSeek-R1-0528 across its evaluations when quantized to NVFP4 with post-training quantization[43]. That is a vendor result for one very large model; validate smaller models yourself.

GGUF

GGUF is llama.cpp's file format and a family of block-quantization schemes. The llama.cpp documentation lists bits per weight from about 2.0 (IQ1_S) to 8.5 (Q8_0), and gives a 70B model as 280.9 GB at its original size versus 43.1 GB at Q4_K_M[45]. Importance-matrix (--imatrix) calibration improves low-bit quality. For GPU servers, prefer formats with native vLLM or SGLang kernels.

Accuracy Tradeoffs

In practice, 8-bit is close to lossless for most modern LLMs; 4-bit weight-only is usually fine for chat and summarization but degrades first on math, code and long reasoning; below 4 bits quality drops quickly. Evaluate on a task that resembles the research use, not only a generic leaderboard.

Field note: For reproducible research, record the model repository and revision hash, the quantization method, the engine version and the sampling parameters. "We used Llama 70B" is not a method; a quantized checkpoint served by a different engine version can give measurably different outputs.

Multi-GPU Inference and Sizing

Parallelism for Inference

When a model or its KV cache does not fit on one GPU, or one GPU is too slow for your latency target, split the work. The parallelism types match training (see Training at Scale), but the tradeoffs differ.

• Tensor parallelism (TP) splits each layer's matrices across GPUs, with an all-reduce after each block. It cuts per-GPU weight reads, so it lowers decode latency, but it needs fast links. vLLM's guidance: use TP across the GPUs within a node[13].
• Pipeline parallelism (PP) assigns contiguous layers to different GPUs. It communicates less, tolerates slower links and uneven splits, and suits multi-node deployments: vLLM suggests TP equal to GPUs per node and PP equal to the number of nodes, and recommends PP over TP on GPUs without NVLink, such as the L40S[13].
• Data parallelism (DP) runs independent replicas: more throughput, not more capacity per request. Several small replicas often beat one TP=8 instance.
• Expert parallelism (EP) places different experts of a mixture-of-experts (MoE) model on different GPUs, with all-to-all communication to route tokens.

Serving MoE Models

MoE models activate only a fraction of their parameters per token (DeepSeek-V2: 236B total, 21B active[12]), so compute per token is low but all weights must still be resident. They need the memory of a large model and the bandwidth profile of a small one, which favors big aggregate HBM and fast scale-up domains. vLLM's recommended pattern for large MoE models, especially MLA models such as DeepSeek, is data parallel attention combined with expert (or tensor) parallel MoE layers, enabled with --data-parallel-size and --enable-expert-parallel[49]. The largest deployments span rack-scale NVLink domains; NVIDIA's GB300 NVL72 offers a 40 TB combined HBM and Grace memory budget per rack[43]. See Accelerators and Interconnects for the hardware.

Sizing: A 70B Model in BF16, FP8 and INT4

Size weights first, then KV cache for target concurrency and context, then headroom for activations and CUDA graphs. vLLM claims 92% of GPU memory by default (--gpu-memory-utilization 0.92) and fills what remains after weights and activations with KV cache[19].

Step 1: Weights

Llama 3.3 70B has 70,553,706,496 parameters according to its Hugging Face metadata[50]. Weight memory is parameters times bytes per parameter:

Computed values in this table are parameter count[50] x bytes per parameter; the GGUF row is quoted directly.

Step 2: Which GPU Configurations Fit

Per-GPU memory from vendor pages: H100 SXM 80 GB[5], H200 141 GB[6], B200 180 GB (DGX B200: 1,440 GB across 8 GPUs)[51], AMD MI355X 288 GB[44], L40S 48 GB[52]. "Usable" below is 0.92 x total GPU memory minus weights; activations and CUDA graphs come out of that before KV cache.

Memory figures from the vendor sources cited above; "left" values are arithmetic from those figures and the weight sizes in Step 1.

Step 3: Turn Leftover Memory into Users

Take the single H200 with FP8 weights: about 59 GB remain. At 320 KiB per token with a BF16 cache, that is roughly 180,000 tokens of KV cache before activations are subtracted: about 20 concurrent 8K-token conversations, or one 128K-token one. An FP8 cache doubles that to roughly 360,000 tokens. If your SLO calls for 50 concurrent 8K conversations (about 410,000 tokens), even the FP8 cache falls short, so plan on a second replica or TP=2. Then check speed: one H200 with FP8 weights tops out near 68 tokens/s per user (above); TP=2 roughly halves the per-GPU weight read.

Field note: The most common sizing mistake is setting --max-model-len to the model's full advertised context (often 128K) "just in case". It rarely causes a crash, but it lets a few users with huge prompts consume the whole cache while everyone else queues. Set the context limit to what your users actually need, and offer a separate long-context endpoint if some projects require it.

Embeddings, RAG and Vector Databases

Retrieval-augmented generation (RAG) indexes a corpus (papers, lab notebooks, policies, code), retrieves relevant passages per query and has the LLM answer from them. MLPerf Inference v6.1 added an end-to-end RAG benchmark that captures the full pipeline: an embedding model turns the query into a vector, a retriever pulls candidates from a vector database, a re-ranker refines them, and one or more LLMs produce the answer[40].

Embedding and Re-ranking Models

Embedding models are encoders: they do one prefill-like pass per input and no decode, so they are compute bound and batch extremely well. vLLM serves them as "pooling" models through LLM.embed and LLM.score offline and through OpenAI-compatible embedding endpoints online[32]; llama.cpp's server also offers an embeddings route[30]. Corpus ingestion is a batch job; the query path is a small service that can often share a GPU (see Scheduling and Orchestration).

Vector Databases

You do not always need a dedicated vector database:

• Faiss is a library, not a server: a toolkit of indexing methods for vector similarity search, clustering and compression, with GPU support[53]. Ideal for research code and offline experiments.
• pgvector adds vector columns and indexes (HNSW and IVFFlat) to PostgreSQL. HNSW gives a better speed-recall tradeoff than IVFFlat at the cost of slower builds and more memory[54]. If your group already runs Postgres, start here.
• Milvus is a dedicated open-source vector database under the LF AI and Data Foundation, with deployment modes from a laptop to Kubernetes clusters handling billions of vectors[55].

Access control is the part people forget: a RAG index over restricted documents exposes them to anyone who can query it. Mirror source permissions in retrieval, and treat indexes of regulated data as regulated systems (see Research Computing Operations).

Offline Batch Inference vs. Interactive Serving

Labeling a corpus, generating synthetic data or scoring evaluation outputs are batch jobs with no latency SLO. Treat them as HPC jobs:

• Use the offline API, not an HTTP server. vLLM's LLM class takes a whole list of prompts and schedules them with continuous batching internally[32]. No ports, no API keys, no idle GPU between requests.
• Tune for throughput. Large max_num_seqs, prefix caching on for shared instructions, and no speculative decoding at high batch sizes.
• Shard the data, not the model. Ten single-GPU array tasks each processing one tenth of the input are simpler and more robust than one ten-GPU job, provided the model fits on one GPU.
• Checkpoint output. Write results incrementally so a preempted job resumes instead of restarting.

Interactive serving is long-running, multi-tenant and networked; it needs authentication, monitoring and an owner. A pragmatic split is a small pool of dedicated inference nodes (or a Kubernetes partition) for services, and ordinary batch partitions for offline work.

Example: Slurm Batch Inference Job

This job runs an FP8 70B model with TP=2 over a JSONL file of prompts. Partition, module and path names are site-specific examples; the Slurm options are standard[56] and the vLLM calls use the documented offline API[32].

#!/bin/bash
#SBATCH --job-name=llm-batch
#SBATCH --partition=gpu          # example partition name
#SBATCH --nodes=1
#SBATCH --gpus-per-node=2        # e.g. 2x H100 80GB for FP8 70B at TP=2
#SBATCH --cpus-per-task=16
#SBATCH --mem=256G
#SBATCH --time=08:00:00
#SBATCH --array=0-9              # 10 shards of the input
#SBATCH --output=logs/%x_%A_%a.out

set -euo pipefail
module load cuda                 # site-specific; or use a container
source "$HOME/venvs/vllm/bin/activate"

export HF_HOME=/scratch/$USER/hf_cache   # keep model downloads off $HOME
export OMP_NUM_THREADS=8

python batch_infer.py \
  --model RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --input  data/prompts.jsonl \
  --output results/out_${SLURM_ARRAY_TASK_ID}.jsonl \
  --shard  "${SLURM_ARRAY_TASK_ID}" \
  --num-shards 10 \
  --tp 2

# batch_infer.py - offline batch inference with vLLM (example)
import argparse, json
from vllm import LLM, SamplingParams

p = argparse.ArgumentParser()
p.add_argument("--model", required=True)
p.add_argument("--input", required=True)
p.add_argument("--output", required=True)
p.add_argument("--shard", type=int, default=0)
p.add_argument("--num-shards", type=int, default=1)
p.add_argument("--tp", type=int, default=1)
args = p.parse_args()

with open(args.input) as f:
    rows = [json.loads(line) for i, line in enumerate(f)
            if i % args.num_shards == args.shard]

llm = LLM(model=args.model,
          tensor_parallel_size=args.tp,
          max_model_len=8192,            # size to your prompts, not the model max
          gpu_memory_utilization=0.92)

params = SamplingParams(temperature=0.0, max_tokens=512)
conversations = [[{"role": "user", "content": r["prompt"]}] for r in rows]

outputs = llm.chat(conversations, params)

with open(args.output, "w") as out:
    for r, o in zip(rows, outputs):
        out.write(json.dumps({"id": r["id"], "response": o.outputs[0].text}) + "\n")

OpenAI-Compatible APIs and Campus AI Gateways

The OpenAI chat-completions API is the de facto LLM wire protocol. vLLM, SGLang, llama.cpp, Ollama and Ray Serve LLM all expose it[57][58][30][31][27], so scripts switch between local, campus and commercial models by changing a base URL and key.

Example: Launching a vLLM Server

All flags below exist in current vLLM[17][57]; values are examples for one H200 serving the FP8 70B model from the sizing section.

# Example: FP8 Llama 3.3 70B on a single H200 (141 GB)
export HF_HOME=/scratch/models/hf_cache

vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --served-model-name llama-3.3-70b \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 64 \
  --enable-prefix-caching \
  --host 0.0.0.0 --port 8000 \
  --api-key "$(cat /etc/vllm/api_key)"

# Smoke test
curl -s http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer $(cat /etc/vllm/api_key)" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.3-70b",
       "messages": [{"role": "user", "content": "Say hello in five words."}]}'

Prefix caching is already on by default, so that flag documents intent[19]. Check the startup KV cache and concurrency lines before opening the service.

Campus AI Gateways

Once you have more than one model or backend, put a gateway in front. LiteLLM's proxy presents one OpenAI-compatible endpoint in front of 100+ LLMs, including self-hosted OpenAI-compatible servers such as vLLM, with per-key budgets, rate limits and spend tracking[59]. A minimal config routes a public model name to the vLLM server above:

# litellm config.yaml (example)
model_list:
  - model_name: llama-3.3-70b            # name clients request
    litellm_params:
      model: openai/llama-3.3-70b        # "openai/" = OpenAI-compatible backend
      api_base: http://gpu-node17:8000/v1
      api_key: "os.environ/VLLM_API_KEY"
      rpm: 600                           # example per-deployment rate limit
  - model_name: llama-3.3-70b            # second replica, load-balanced
    litellm_params:
      model: openai/llama-3.3-70b
      api_base: http://gpu-node18:8000/v1
      api_key: "os.environ/VLLM_API_KEY"

general_settings:
  master_key: sk-change-me               # example only; keep real keys in a secret store

Start it with litellm --config config.yaml[60]. A gateway is one place to issue per-lab keys tied to campus identity, enforce quotas, log usage for allocation reports, route between on-prem and commercial models, and apply data-classification rules. See Research Computing Operations and Cloud and Hybrid AI/HPC.

Field note: Never expose a raw engine port to the campus network. Put a gateway or reverse proxy in front that handles TLS, authentication and logging, and bind the engine to localhost or a private VLAN. Engine API keys are a single shared secret, not a user management system.

Benchmarking, Cost and Energy

Cost and Energy per Token

Budget in cost or energy per token at your SLO, not per GPU-hour. Published data points:

• Google measured the median Gemini Apps text prompt at 0.24 Wh of energy, counting accelerators, host systems, idle capacity and data-center overhead, and reported that software efficiency efforts and clean energy procurement drove a 33x reduction in energy per median prompt over one year[61].
• The ML.ENERGY benchmark measured Qwen 3 32B on one B200 at 0.151 J per token for text conversation at its maximum batch size, versus 0.209 J at batch 128. Reasoning traffic cost 0.312 J per token because longer outputs consumed KV memory and capped the batch at 128. Per response, the gap was 95 J versus 2,192 J (23x)[2].
• NVIDIA claims up to 25x (Blackwell) and 50x (Blackwell Ultra) better energy efficiency per token than H100 with FP4; treat it as a vendor peak comparison[43].

The lessons: energy per token falls sharply with batch size, so a lightly loaded dedicated GPU is the least efficient way to serve; reasoning and agent workloads cost far more per request than chat. For your own numbers, measure node power (not just GPU board power) during a benchmark and divide by tokens generated. For the facility side, see Power, Cooling and Facilities.

Benchmarking: MLPerf and Engine Tools

MLPerf Inference

MLPerf Inference is the peer-reviewed, architecture-neutral benchmark for inference systems. The v6.1 round, published September 16, 2026, drew a record 30 submitting organizations, added the end-to-end RAG and Edge Agentic Inference tests, allowed speculative decoding in the interactive scenario, and included the largest system yet submitted (512 accelerators). New hardware included AMD Instinct MI350P, AMD Ryzen AI Max+ 395 and Intel Arc Pro B70 as available products, and NVIDIA Rubin and Vera Rubin NVL72 in preview. The best per-accelerator DeepSeek-R1 server result improved 5.7x over v5.1 a year earlier[40]. The v6.1 datacenter suite includes Llama 2 70B, Llama 3.1 8B, DeepSeek-R1, GPT-OSS 120B, a Qwen3-VL vision-language model, Whisper, and the new RAG test[62]. MLCommons has announced that an API-centric suite, MLPerf Endpoints, will replace MLPerf Inference for the datacenter[40].

Use MLPerf to compare systems for procurement, noting "available" versus "preview" status; it does not replace testing your own model and traffic.

Engine Benchmark Tools

For your own measurements, drive the real endpoint with realistic load:

• vllm bench serve works against any OpenAI-compatible server (not only vLLM), with synthetic or dataset-based prompts, request-rate or concurrency control, percentile reporting and goodput SLOs[8].
• AIPerf is NVIDIA's current tool; its predecessor GenAI-Perf is being phased out[63].

# Example: 500 requests, 2K-token prompts, 256-token outputs, 64 concurrent users,
# goodput counted against TTFT <= 2 s and TPOT <= 50 ms
vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --served-model-name llama-3.3-70b \
  --dataset-name random \
  --random-input-len 2048 --random-output-len 256 \
  --num-prompts 500 \
  --max-concurrency 64 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --goodput ttft:2000 tpot:50 \
  --save-result

Sweep concurrency (1, 8, 32, 64, 128) and plot throughput against p99 TTFT and TPOT; the knee of that curve is your practical capacity per replica.

Key Takeaways

• Inference is a recurring cost that now dominates AI compute; decode is bound by memory bandwidth, so bytes moved per token, not FLOPS, set single-user speed.
• Track TTFT, TPOT/ITL and goodput against written SLOs; raw throughput numbers without latency targets are misleading.
• The KV cache (2 x layers x kv_heads x head_dim x seq x bytes) limits concurrency; GQA, MLA, FP8 KV cache and sensible context limits are the main levers.
• Start with vLLM or SGLang; add Dynamo, Ray Serve or disaggregation only when a measured bottleneck justifies it. Migrate off TGI, which is archived.
• FP8 is the near-lossless default on Hopper-class and newer GPUs; INT4 AWQ/GPTQ opens older GPUs; NVFP4 is the Blackwell path. Evaluate on your own tasks.
• Size in three steps: weights, KV cache for target concurrency, and headroom. A 70B model is one H200 in FP8, two H100s in FP8, or one H100 in INT4.
• Run research batch inference as ordinary Slurm jobs with the offline API; run interactive services behind an authenticated OpenAI-compatible gateway such as LiteLLM.
• Benchmark your own model and traffic with vllm bench serve or AIPerf, and use MLPerf results for cross-vendor comparison, not as a sizing answer.

Sources

1. Patterson et al., The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink (arXiv, 2022)
2. Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (ML.ENERGY Initiative, 2026)
3. Mastering LLM Techniques: Inference Optimization (NVIDIA Technical Blog, 2023)
4. Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024, arXiv)
5. NVIDIA H100 Tensor Core GPU product page and specifications (NVIDIA)
6. NVIDIA H200 GPU product page and specifications (NVIDIA)
7. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024, arXiv)
8. vllm bench serve CLI reference (vLLM documentation, 2026)
9. Llama-3.3-70B-Instruct-FP8-dynamic config.json (Red Hat AI on Hugging Face)
10. Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (arXiv, 2019)
11. Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023, arXiv)
12. DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv, 2024)
13. Parallelism and Scaling (vLLM documentation, 2026)
14. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (USENIX OSDI, 2022)
15. TensorRT LLM Overview (NVIDIA documentation, 2026)
16. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023, arXiv)
17. vLLM engine argument definitions, vllm/engine/arg_utils.py (vLLM GitHub, 2026)
18. Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (arXiv, 2023)
19. vLLM CacheConfig definitions, vllm/config/cache.py (vLLM GitHub, 2026)
20. vLLM releases, including v0.29.0 release notes (vLLM GitHub, 2026)
21. vLLM v0.30.0 release (vLLM GitHub, September 2026)
22. SGLang v0.5.20 release (SGLang GitHub, September 2026)
23. SGLang documentation home (LMSYS, 2026)
24. Triton Inference Server README (NVIDIA GitHub, 2026)
25. NVIDIA Dynamo README (ai-dynamo GitHub, 2026)
26. Dynamo v1.5.0 release notes (ai-dynamo GitHub, September 2026)
27. Serving LLMs with Ray Serve LLM (Ray documentation, 2026)
28. Text Generation Inference README, maintenance-mode notice (Hugging Face GitHub)
29. GitHub API repository metadata for huggingface/text-generation-inference, archived flag (GitHub, 2026)
30. llama.cpp server README (ggml-org GitHub, 2026)
31. OpenAI compatibility (Ollama documentation, 2026)
32. Offline Inference (vLLM documentation, 2026)
33. Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv, 2024)
34. Disaggregated Serving (NVIDIA Dynamo documentation, 2026)
35. Disaggregated Prefilling (experimental) (vLLM documentation, 2026)
36. Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding (ICML 2023, arXiv)
37. Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (arXiv, 2024)
38. Li et al., EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (arXiv, 2025)
39. Speculative Decoding (vLLM documentation, 2026)
40. MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results (MLCommons, September 2026)
41. Quantization (vLLM documentation, 2026)
42. FP8 W8A8 (vLLM documentation, 2026)
43. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (NVIDIA Technical Blog, 2025)
44. AMD Instinct MI355X GPU specifications (AMD)
45. llama.cpp quantize README (ggml-org GitHub, 2026)
46. Llama-3.3-70B-Instruct-FP8-dynamic model card (Red Hat AI on Hugging Face, 2024)
47. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv, 2022)
48. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv, 2023)
49. Data Parallel Deployment (vLLM documentation, 2026)
50. Hugging Face model API metadata for meta-llama/Llama-3.3-70B-Instruct, safetensors parameter total (Hugging Face)
51. Introduction to NVIDIA DGX B200 Systems (NVIDIA DGX B200 User Guide)
52. NVIDIA L40S GPU product page and specifications (NVIDIA)
53. Douze et al., The Faiss library (arXiv, 2024)
54. pgvector README (pgvector GitHub, 2026)
55. What is Milvus? (Milvus documentation, 2026)
56. sbatch manual page (SchedMD)
57. vllm serve CLI reference (vLLM documentation, 2026)
58. Tutorial: Sending a request (SGLang documentation, 2026)
59. LiteLLM AI Gateway (LLM Proxy) (LiteLLM documentation, 2026)
60. LiteLLM proxy config.yaml reference (LiteLLM documentation, 2026)
61. Elsworth et al., Measuring the environmental impact of delivering AI at Google Scale (arXiv, 2025)
62. MLPerf Inference Benchmark Suite, v6.1 model list (MLCommons documentation, 2026)
63. GenAI-Perf README, phase-out notice pointing to AIPerf (NVIDIA documentation, 2026)