Observability, Performance and Efficiency

A GPU cluster costs far more per hour than the CPU clusters this manual was first written for, and it fails in more interesting ways. The old habit of watching load averages in Ganglia and reading the Slurm logs when someone complains will not cut it. You need to know, per job and per GPU, whether the silicon is doing useful work, how much energy it burned, and whether a marginal GPU, cable or NIC is dragging a thousand-GPU job to a crawl. This chapter covers the telemetry stack (driver tools, DCGM, Prometheus and Grafana), job-level accounting in Slurm and Open XDMoD, node and fabric health, the profilers you will actually use, the efficiency metrics that matter for AI and HPC, power capping and energy accounting, the standard benchmarks, and what reliability looks like at scale. Facilities-side metering (PUE, rack power) is covered in Power, Cooling and Facilities; scheduler configuration is in Scheduling and Orchestration.

GPU Telemetry: The Ground Truth

Every useful observability stack starts with the vendor management library on each node: NVML for NVIDIA GPUs, AMD SMI (the library behind amd-smi) for AMD GPUs. The command-line tools are thin wrappers around those libraries, and so are the Prometheus exporters and the Slurm plugins.

nvidia-smi

nvidia-smi is the first thing to run on any NVIDIA node. Beyond the default summary, three modes matter for operations: machine-readable queries, streaming device monitoring, and detailed per-subsystem reports.

# CSV snapshot every 5 seconds: utilization, memory, power, temperature
nvidia-smi --query-gpu=timestamp,index,utilization.gpu,utilization.memory,memory.used,power.draw,temperature.gpu \
           --format=csv -l 5

# Stream power/temp (p), utilization (u), clocks (c), violations (v), memory (m)
nvidia-smi dmon -s pucvm

# Detailed reports for ECC, power, clock event reasons and row remapping
nvidia-smi -q -d ECC,POWER
nvidia-smi -q -d PERFORMANCE
nvidia-smi -q -d ROW_REMAPPER

The --query-gpu field names above appear in vendor documentation[1]; the dmon -s metric groups and the -q -d display sections are documented in the nvidia-smi manual.[2] Run nvidia-smi --help-query-gpu for the full field list on your driver version.

Understand what "utilization" means before you put it on a dashboard. The nvidia-smi documentation defines GPU utilization as the "percent of time over the past sample period during which one or more kernels was executing on the GPU", with a sample period between 1 second and 1/6 second depending on the product.[2] A single tiny kernel that occupies one SM out of more than a hundred still counts as 100% for that interval. We return to this in Metrics That Matter.

rocm-smi and amd-smi

On AMD Instinct systems, amd-smi is the current tool and rocm-smi is on its way out. AMD's own tooling notes that rocm-smi is deprecated in ROCm 7.0[3], and AMD positions amd-smi as its replacement, with amd-smi list, amd-smi monitor and amd-smi static covering the common rocm-smi tasks.[4] Useful commands from the AMD SMI CLI documentation:[5]

# Live table, refreshed every second
amd-smi monitor --watch 1

# Selected columns: power and temperature; ECC and VRAM
amd-smi monitor --power-usage --temperature
amd-smi monitor --vram-usage --ecc

# Point-in-time metrics: engine usage, power, ECC totals
amd-smi metric --usage --power
amd-smi metric --ecc

# Retired memory pages (a leading indicator of failing HBM)
amd-smi bad-pages --retired

# Machine-readable output for scripts
amd-smi metric --usage --power --json

AMD's documentation warns that fields sourced from the driver's versioned gpu_metrics structure read N/A when the driver is newer than the installed AMD SMI release; if power or usage suddenly shows N/A after a driver update, upgrade AMD SMI or ROCm to match.[5]

DCGM and dcgm-exporter

NVIDIA Data Center GPU Manager (DCGM) is the fleet-grade layer: a host engine (nv-hostengine) that watches fields continuously, a CLI (dcgmi), health checks, active diagnostics, per-job statistics and low-overhead hardware profiling counters.[6] The profiling fields are what make DCGM worth deploying even if you already scrape nvidia-smi. By default DCGM samples them at 1 Hz, with a minimum configurable interval of 100 ms.[6]

# Which profiling fields does this GPU support, and which can be collected together?
dcgmi profile --list --entity-id gpu:0

# Stream SM activity (1002) and DRAM activity (1005) once per second
dcgmi dmon --entity-id gpu:0 --field-id 1002,1005 --delay 1000

# Graphics engine active (1001), tensor pipe active (1004), DRAM active (1005)
dcgmi dmon -e 1001,1004,1005

DCGM's documentation defines SM activity as "the fraction of time at least one warp was active on an SM, averaged over all SMs", and DRAM activity as the fraction of cycles with device-memory traffic.[6] Hardware limits mean some counters cannot be collected together; DCGM multiplexes automatically, and sampling too fast while multiplexing can return zeroes.[6]

dcgm-exporter publishes DCGM fields in Prometheus text format. NVIDIA's install guide gives the container quick start (the SYS_ADMIN capability is needed for the DCGM_FI_PROF_* profiling fields), and the default listener is port 9400.[7][8]

docker run -d --rm --name dcgm-exporter \
  --gpus all --cap-add SYS_ADMIN -p 9400:9400 \
  nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}

curl --fail http://localhost:9400/metrics | grep -E 'DCGM_FI_(DEV_GPU_UTIL|PROF_GR_ENGINE_ACTIVE|DEV_POWER_USAGE)'

The shipped default collector list includes clocks, temperatures, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION (a counter in millijoules since driver load), DCGM_FI_DEV_GPU_UTIL, framebuffer usage, NVLink bandwidth, row-remap counters, Xid errors, and the profiling fields DCGM_FI_PROF_GR_ENGINE_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE and DCGM_FI_PROF_DRAM_ACTIVE.[9] Note that DCGM_FI_PROF_SM_ACTIVE ships commented out; enable it with a custom collector CSV passed via --collectors.[9][7] For HPC sites the most useful option is --hpc-job-mapping-dir, which reads scheduler-created GPU-to-job mapping files and adds an hpc_job label to each GPU's metrics.[8][10] That one label turns a node dashboard into a job dashboard.

Field note: DCGM profiling counters and developer profilers (Nsight Systems, Nsight Compute) compete for the same hardware. When a user needs to profile, run dcgmi profile --pause on the node and dcgmi profile --resume afterward. Pause is host-engine-wide, and paused fields read as blank, so make sure your dashboards and alerts tolerate gaps.[6]

Prometheus and Grafana

The de facto open stack is Prometheus scraping exporters on every node, Grafana for dashboards, and Alertmanager for paging. A minimal scrape configuration for GPU nodes (hostnames are examples):

# prometheus.yml (excerpt) - example hostnames
scrape_configs:
  - job_name: dcgm
    scrape_interval: 30s
    static_configs:
      - targets:
          - gpu001:9400
          - gpu002:9400
  - job_name: node
    scrape_interval: 30s
    static_configs:
      - targets:
          - gpu001:9100
          - gpu002:9100

The node job scrapes a host-metrics exporter such as node_exporter (the port shown is an example; use whatever your exporter listens on). dcgm-exporter's own default collection interval is 30,000 ms, so scraping faster than that just returns repeated samples unless you lower --collect-interval.[8] NVIDIA publishes a Grafana dashboard for dcgm-exporter (Grafana dashboard 12239) and a checked-in JSON version in the repository.[10]

Useful PromQL starting points, assuming --hpc-job-mapping-dir is enabled so metrics carry an hpc_job label:

# Average graphics-engine activity per job over the last hour
avg by (hpc_job) (avg_over_time(DCGM_FI_PROF_GR_ENGINE_ACTIVE[1h]))

# Allocated GPUs that have been essentially idle for an hour (utilization below 5%)
avg_over_time(DCGM_FI_DEV_GPU_UTIL{hpc_job!=""}[1h]) < 5

# Energy per job over the last 24 hours, in kWh (counter is in millijoules)
sum by (hpc_job) (increase(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION[24h])) / 3.6e9

Treat the thresholds as examples to tune for your workload mix. Keep long-term storage in mind: per-GPU metrics at 30-second resolution add up quickly on a large cluster, and most sites downsample or push to a long-term store (Thanos, Mimir, VictoriaMetrics or similar) for year-over-year reporting.

Job-Level Accounting

Node dashboards tell you the cluster is busy. Job accounting tells you who used it, how well, and what it cost. For the scheduler side of this, see Scheduling and Orchestration.

Slurm accounting for GPUs

Slurm tracks consumable resources as TRES. Adding gres/gpu to AccountingStorageTRES records GPU allocations, and per the Slurm documentation also enables gres/gpumem and gres/gpuutil.[11] Those two utilization TRES are only available for NVIDIA GPUs with AutoDetect=nvml and AMD GPUs with AutoDetect=rsmi, and are not provided for MIG devices because NVML does not support utilization metrics for MIG.[12]

# slurm.conf (excerpt)
AccountingStorageTRES=gres/gpu
JobAcctGatherType=jobacct_gather/cgroup

# gres.conf (excerpt)
AutoDetect=nvml

# GPUs allocated, elapsed time and energy for a user's jobs since Monday
sacct -u alice -S 2026-09-21 -X \
      --format=JobID,JobName%20,Elapsed,AllocTRES%60,ConsumedEnergy,State

# Per-step GPU memory high-water mark and utilization averaged across tasks
sacct -j 1277.0 --format=TRESUsageInAve -p
sacct -j 1277.0 --format=TRESUsageInTot -p

# Cluster-wide GPU hours by account for last month
sreport cluster AccountUtilizationByUser start=2026-08-01 end=2026-09-01 --tres=gres/gpu

The Slurm GRES documentation shows exactly this kind of output, for example gres/gpumem=36266M,gres/gpuutil=100 in TRESUsageInAve.[12] ConsumedEnergy is reported in joules.[13] Use sreport with --tres= to report GPU hours rather than CPU hours.[11]

Open XDMoD with GPU metrics

Open XDMoD, from the University at Buffalo's Center for Computational Research, is the standard tool for center-level usage reporting in the US academic community. It has tracked GPUs in its Jobs realm since version 9.0.0, taking the GPU count for Slurm jobs from the gres/gpu entry of the AllocTRES accounting field; if GPU data is missing or unparsable it reports zero GPU hours.[14] The Job Performance (SUPReMM) module adds per-job performance data: it gained GPU usage and GPU memory metrics in version 8.1.0 and an Efficiency tab with total GPU usage statistics in version 10.0.0.[15] That combination (allocation from Slurm, actual usage from node telemetry) is what lets you tell a PI "your group used 40,000 GPU hours and the median job kept its GPUs busy X% of the time".

Field note: The Open XDMoD documentation carries a security warning about vulnerabilities in versions 11.0.3 and earlier and recommends upgrading to 11.0.4.[14] Usage portals often sit on a web server nobody patches after the grant ends; put yours on the same patch cycle as everything else.

DCGM job statistics

DCGM can record per-job GPU statistics bracketed by scheduler prolog and epilog scripts: create a GPU group, enable stats, start recording with the job ID as the label, stop at job end, and retrieve a summary.[16][17]

# One-time: create a group of GPUs and enable watches (group ID returned is used below)
dcgmi group -c allgpus -a 0,1,2,3
dcgmi stats -g 1 --enable

# Prolog: start recording for this job
dcgmi stats -g 1 -s "$SLURM_JOB_ID"

# Epilog: stop recording and dump a verbose report
dcgmi stats -x "$SLURM_JOB_ID"
dcgmi stats -j "$SLURM_JOB_ID" -v > /var/log/dcgm-jobs/"$SLURM_JOB_ID".txt

Note that in dcgmi stats, -j means --job, not JSON output.[17] NVIDIA's example report includes energy consumed in joules, average power, and average SM and memory utilization.[16]

Node Health Checks

The cheapest outage is the one where a bad node drains itself before a job lands on it.

LBNL Node Health Check (NHC)

NHC is the long-standing framework for periodic node checks under Slurm and other schedulers. Its README gives the Slurm integration directly:[18]

# slurm.conf
HealthCheckProgram=/usr/sbin/nhc
HealthCheckInterval=300

# /etc/nhc/nhc.conf (examples; adjust to your hardware)
* || check_fs_mount_rw /
* || check_fs_mount_rw /scratch
* || check_ps_service -u root -S sshd
* || check_hw_ib 400

check_hw_ib verifies an active InfiniBand link at the specified rate in Gb/s (the README's own example is check_hw_ib 40 for QDR, which dates the project nicely); set the value to your actual link rate.[18] NHC's built-in GPU check targets the old nvidia-healthmon tool, so for current GPUs call DCGM from a custom check or a separate script.[18]

DCGM diagnostics

dcgmi diag runs active tests at four levels: level 1 is quick deployment checks lasting seconds, level 2 is medium validation of about two minutes, level 3 is long hardware diagnostics of about 15 minutes, and level 4 is extended. The aliases quick, short, medium, long and xlong map to these levels.[19] Diagnostics consume substantial GPU, power and fabric resources, so never run level 3 or 4 on a node with a user job.[19]

#!/bin/bash
# gpu-quickcheck.sh - example health gate for epilog or cron; drains node on failure
NODE=$(hostname -s)
if ! dcgmi diag -r 1 -j > /tmp/dcgm-diag.json 2>&1; then
    scontrol update NodeName="$NODE" State=DRAIN Reason="dcgmi diag r1 failed"
    exit 1
fi
# Fail if the expected number of GPUs is not visible (8 is an example)
if [ "$(nvidia-smi -L | wc -l)" -ne 8 ]; then
    scontrol update NodeName="$NODE" State=DRAIN Reason="GPU count mismatch"
    exit 1
fi

dcgmi diag also supports --expectedNumEntities gpu:N to fail fast when a GPU has fallen off the bus, and --fail-early for long stress tests.[19] A common pattern is level 1 in the epilog after every job, level 3 weekly or after any hardware service, and level 4 during acceptance testing.

Fabric Monitoring

At scale, the network is where performance problems hide. A single link flapping or running at high bit error rate slows every collective that crosses it. For fabric design see Interconnects.

UFM and InfiniBand telemetry

NVIDIA Unified Fabric Manager (UFM) Telemetry collects over 120 counters per port across an InfiniBand fabric, including bit error rate, temperature, histograms and retransmissions, and NVIDIA's cluster bring-up procedure uses it to find marginal cables before they fail.[20] The bring-up guide shows pulling link-quality counters as CSV (the IP address is a placeholder):

curl http://{machine_ip}:9002/csv/xcset/low_freq_debug >> my_telemetry_file.csv

The same guide gives evaluation criteria: physical link state must be up, post-FEC and symbol error rates are compared against normal/warning/error thresholds that depend on cable type and FEC mode, any increase in the involuntary link-down counter should be traced, and two link-downs on the same port within 24 hours call for investigation. It also notes a minimum port up time of 125 minutes for a valid BER measurement.[20]

Host-side counters in job accounting

Slurm can attribute InfiniBand traffic to jobs with AcctGatherInterconnectType=acct_gather_interconnect/ofed, which provides PacketsIn, PacketsOut, InMB and OutMB per job.[21] This is coarse, but it quickly answers "is this job actually using the fabric, or is it eight independent single-node jobs pretending to be one?"

Collective benchmarks as health checks

The simplest end-to-end fabric check is a collective benchmark. NVIDIA's nccl-tests give the canonical commands:[22]

# Single node, 8 GPUs, message sizes from 8 bytes to 128 MiB
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8

# 64 GPUs across 8 nodes (binaries built with MPI=1)
mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Record the bus bandwidth ("busbw") at large message sizes for every node pair or rack at acceptance, then rerun after maintenance. A node or rail that falls well below its peers is your straggler candidate. AMD's equivalent is rccl-tests.

Profilers: From Timeline to Kernel

Telemetry tells you that a job is inefficient. Profilers tell you why. Use them in order: a timeline tool first to find where time goes, then a kernel tool on the few kernels that matter. DCGM's own documentation recommends exactly this workflow: locate an underused device with DCGM, then use Nsight Systems or Nsight Compute for attribution.[6]

Nsight Systems

Nsight Systems produces a system-wide timeline of CPU threads, CUDA API calls, kernels, memory copies, NVTX ranges, MPI and more. Typical use under Slurm:

srun -n 8 nsys profile -t cuda,nvtx,mpi --stats=true \
     -o /scratch/$USER/nsys/run_%q{SLURM_PROCID} python train.py

The -t cuda,nvtx,mpi trace selection appears in NVIDIA's own Slurm container example, and --stats=true generates summary statistics (and an SQLite database) after collection.[23] The user guide notes that Nsight Systems does not officially support runs longer than 5 minutes[23], so for long training jobs use --capture-range with the CUDA profiler API or NVTX to capture a handful of steady-state iterations rather than the whole run.

Nsight Compute

Nsight Compute (ncu) profiles individual kernels in depth: occupancy, memory throughput, instruction mix, roofline position. It replays kernels to collect counters, so it is slow; always filter.

# Full metric set for one launch of kernels matching a regex
ncu --set full -k regex:gemm -c 1 -o gemm_report ./app

# MPI: one report per rank
mpirun -np 4 ncu -o report_%q{OMPI_COMM_WORLD_RANK} ./app

If no --set is given, ncu collects the basic set; --set full collects all sections. -k filters by kernel name and -c limits the launch count.[24]

rocprofv3

On AMD GPUs, rocprofv3 (from ROCprofiler-SDK) is the current command-line tracer and profiler. Commands from the ROCm documentation:[25]

# Kernel dispatch trace to CSV
rocprofv3 --kernel-trace --output-format csv -- ./app

# HIP API trace with summary statistics
rocprofv3 --stats --hip-trace --output-format csv -- ./app

# Full system trace viewable in Perfetto
rocprofv3 --sys-trace --output-format pftrace -- ./app

The default output format is a rocpd SQLite database, convertible to CSV, Perfetto or OTF2 with rocpd convert.[25]

PyTorch profiler

For AI workloads, start inside the framework. torch.profiler records operator-level CPU and GPU activity, input shapes and memory, and exports traces for TensorBoard or Perfetto.[26] Use a schedule so you capture a few steady-state steps, not the warm-up:

import torch
from torch.profiler import profile, schedule, ProfilerActivity, tensorboard_trace_handler

prof_schedule = schedule(wait=1, warmup=1, active=3, repeat=1)

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=prof_schedule,
    on_trace_ready=tensorboard_trace_handler("./tb_logs"),
    record_shapes=True,
    profile_memory=True,
) as prof:
    for step, batch in enumerate(loader):
        train_step(model, batch)   # your training step
        prof.step()
        if step >= 6:
            break

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=15))

For training-specific tuning see Training at Scale.

HPCToolkit and TAU

For MPI and OpenMP simulation codes, and for mixed CPU/GPU codes where you need call-path attribution across thousands of ranks, the DOE-supported open tools remain the best option:

• HPCToolkit uses sampling for low overhead and supports CUDA, ROCm and Level Zero. The hpcrun event -e gpu=cuda gives coarse-grain profiling of GPU operations, -e gpu=cuda,pc adds PC sampling inside kernels, and -e gpu=rocm and -e gpu=level0 cover AMD and Intel GPUs. CPU and GPU events can be combined in one run.[27]
• TAU can profile an unmodified binary by prepending tau_exec, which preloads an instrumentation library; OLCF documents tau_exec -T serial,rocm for ROCm and pprof or paraprof for viewing results, with OTF2 trace output for Vampir.[28]

# HPCToolkit: profile GPU operations in an MPI job
srun -n 16 hpcrun -e gpu=cuda ./sim input.nml

# TAU: profile without recompiling, then view
srun -n 4 tau_exec ./sim input.nml
pprof

For guidance on porting and tuning simulation codes see Scientific Computing in the AI Era.

Field note: Most users never profile because nobody showed them how. A one-hour "profile your own job" clinic, where each attendee brings a real job and leaves with an Nsight Systems or PyTorch profiler trace, finds more waste than any dashboard. The usual culprits are data loading on the CPU, host-device copies in the training loop, and tiny kernels launched one at a time.

Metrics That Matter

GPU utilization vs. SM activity

The most common mistake in GPU reporting is treating "GPU utilization" as efficiency. As defined above, it only means some kernel was running.[2] DCGM's profiling fields are more honest:

A job at 100% GPU utilization but 15% SM activity is launching kernels too small to fill the device. A dense-matrix AI training job with low tensor-pipe activity is probably not using mixed precision, or is bottlenecked elsewhere. DCGM itself cautions that no single field proves a job is compute- or memory-bound; compare several and correlate with application throughput.[6]

Model FLOPs utilization (MFU)

For training, the metric that maps directly to money is MFU, introduced in the PaLM paper: the ratio of observed throughput (tokens per second) to the theoretical maximum throughput of the system at peak FLOPs, counting only the FLOPs the model requires for the forward and backward passes, not rematerialization.[29] The PaLM paper's approximation for an N-parameter decoder-only model is 6N matmul FLOPs per token (2N forward, 4N backward), plus an attention term of 12LHQT (layers, heads, head dimension, sequence length).[29]

MFU = (tokens_per_second x (6N + 12LHQT)) / (num_gpus x peak_flops_per_gpu)

Published reference points: PaLM 540B reached 46.2% MFU on 6144 TPU v4 chips (57.8% hardware FLOPs utilization), and the same paper computed 21.3% for GPT-3 and 30.2% for Megatron-Turing NLG.[29] Meta reports 38-43% BF16 MFU for Llama 3 405B pre-training on up to 16K GPUs.[30] Use the dense (not sparse) peak for the precision actually used, and state which one you used; see Accelerators for peak figures and The Convergence for precision.

Inference metrics: tokens/s, TTFT, ITL

For serving, NVIDIA's benchmarking guide defines the standard latency and throughput metrics:[31]

• Time to first token (TTFT): time until the first output token, including tokenization and detokenization. It is dominated by prefill and queueing, and is what users perceive as responsiveness.
• Inter-token latency (ITL), also called time per output token (TPOT): the average time between consecutive tokens, excluding the first. It characterizes the decode phase.
• End-to-end request latency and requests per second.
• Tokens per second: both system-wide and per user. As concurrency rises, total system tokens/s increases while tokens/s per user decreases.

The guide cautions that tools implement these differently, so only compare results when definitions align.[31] Report throughput at a stated latency target (for example, maximum tokens/s with p99 TTFT under a threshold), never throughput alone. See Inference at Scale.

Finding Waste

Every GPU cluster has three kinds of waste: allocated-but-idle GPUs, jobs that run but use a small fraction of the device, and jobs that fail late and must be rerun. You can measure all three.

Idle and low-utilization jobs

# Completed GPU jobs from the last week with their average GPU utilization TRES
sacct -a -S now-7days -E now --state=COMPLETED,TIMEOUT \
      --format=JobID,User,Account,AllocTRES%50,Elapsed,TRESUsageInAve%80 -p \
  | grep 'gres/gpu='

Parse gres/gpuutil out of TRESUsageInAve and rank users by GPU hours spent below a threshold. On the live side, the PromQL query in the Prometheus section finds jobs holding idle GPUs right now. Typical patterns and fixes:

• Interactive sessions left open overnight (Jupyter, VS Code): enforce idle timeouts or short wall-time limits on interactive partitions.
• One GPU busy, the rest idle because the code is not actually multi-GPU: show users per-GPU dashboards and default new users to a single GPU.
• Small models on large GPUs: offer MIG or shared-GPU partitions (see Scheduling and Orchestration).
• Data-starved training: low SM activity with high CPU iowait usually means the input pipeline cannot keep up (see Memory and Storage).

Field note: Publish efficiency, do not police it. A weekly email to each PI showing their group's GPU hours and median GPU activity, next to the cluster median, changes behavior faster than any automated job killer and does not generate angry tickets. Save automatic cancellation for truly idle allocations (for example, zero kernel activity for hours).

Power Capping and Energy Accounting

Setting power limits

Both vendors let administrators cap board power, which is useful when a facility is power-constrained or when a workload is memory-bound and gains little from the last few hundred watts.

# NVIDIA: show limits, then set a cap in watts (example value; must be within the reported min/max; root)
nvidia-smi -q -d POWER
sudo nvidia-smi -i 0 -pl 500

# AMD: show limits, then set a power cap in watts (example value; sudo)
amd-smi static --limit
sudo amd-smi set --gpu 0 --power-cap 450

The nvidia-smi -pl (--power-limit) option takes watts, must be between the reported minimum and maximum limit, requires root, and on multi-component modules accepts a --scope argument for GPU-only or whole-module limits.[2] The amd-smi set --power-cap option takes watts and requires sudo.[5] Neither setting persists across a driver reload or reboot, so apply caps from a boot-time service or the scheduler prolog. At rack scale, NVIDIA's GB300 NVL72 uses programmable power caps plus energy storage to smooth the synchronized power swings of training jobs; see Power, Cooling and Facilities.

Measure before you cap: run the actual workload at several limits and plot time-to-solution and energy-to-solution. Codes limited by memory bandwidth or communication may lose little performance under a moderate cap, while compute-bound codes are more sensitive; only measurement on your workload tells you which case you are in.

Energy per job

There are three practical routes to per-job energy on GPU clusters:

1. Slurm energy accounting. AcctGatherEnergyType=acct_gather_energy/gpu collects energy from the GPU management library, but the Slurm documentation states it is only available for rsmi (AMD) at present. The ipmi plugin collects node-level power from the BMC, and Slurm notes that node-level energy only reflects a job's real consumption when the job has the node exclusively.[32] Results appear in sacct as ConsumedEnergy in joules.[13]
2. DCGM job statistics on NVIDIA systems, recorded via prolog/epilog as shown above, report energy consumed per job in joules.[16]
3. Prometheus. Integrate DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION by the hpc_job label, as in the PromQL example earlier.[9][8]

# slurm.conf (excerpt) - node-level energy from the BMC every 30 s (example interval)
AcctGatherEnergyType=acct_gather_energy/ipmi
AcctGatherNodeFreq=30

GPU-only energy understates the true cost, since CPUs, memory, NICs, fans (on air-cooled nodes) and facility overhead all add to it. For chargeback or carbon reporting, multiply node-level energy by your facility PUE, and say so in the report.

Benchmarking

Benchmarks serve three purposes for a center: acceptance testing (does the system meet the contract), regression testing (did the last driver update break something), and external comparison (TOP500, MLPerf). Know what each one measures.

HPL, HPL-MxP and HPCG

HPL solves a dense system with LU factorization and partial pivoting, and its operation count must be 2/3 n^3 + O(n^2) double-precision operations; the TOP500 rules exclude lower-precision solve-and-refine methods.[33] HPL-MxP exists precisely to measure those methods: factor in low precision, then use an FP64 iterative method such as GMRES with the low-precision LU factors as a preconditioner. A run is invalid if the scaled backward error exceeds 16 or if the iterative method needs more than 50 iterations.[34] HPCG is a conjugate gradient benchmark designed as a complement to HPL.[35]

The June 2026 lists show why you should look at all three. LineShine, the new No. 1 at 2.198 EFlop/s on HPL, also leads HPCG at 22.00 PFlop/s but places fourth on HPL-MxP at 7.92 EFlop/s with a 3.6x speedup, consistent with its CPU-only design; El Capitan records 1.809 EFlop/s on HPL and 17.41 PFlop/s on HPCG.[39] HPCG results roughly two orders of magnitude below HPL are normal, and they are a better predictor of many simulation codes than HPL. For your own acceptance tests, the relative numbers matter more than the absolute: run HPL per node and per rack, flag any node more than a few percent below the median, and keep the results as a baseline.

MLPerf

MLPerf Training measures wall-clock time to train a reference model to a target quality. The current benchmark table lists v6.0 large-language-model benchmarks based on DeepSeek v3, GPT-OSS 20B and Llama 3.1 405B, plus FLUX.1 image generation.[36] MLPerf Inference v6.0 (April 1, 2026) was described by MLCommons as its most significant revision, adding GPT-OSS 120B, an expanded DeepSeek-R1 benchmark with an interactive scenario, DLRMv3, a text-to-video test, a vision-language model test and a YOLOv11 edge test; the largest submitted system had 72 nodes and 288 accelerators.[37] MLPerf results are split into Closed division (same model and optimizer as the reference, for apples-to-apples hardware comparison) and Open division, and into Available, Preview and Research categories.[38] When a vendor quotes MLPerf in a proposal, check the division, category, system size and benchmark version, and ask for the submission link.

Field note: Write benchmark requirements into the RFP as "vendor will demonstrate on the delivered system", not "vendor has published". Include HPL per node, nccl-tests bus bandwidth across the full fabric, a storage benchmark, and one or two of your own users' real applications. Rerun the same suite after every major driver, firmware or OS update.

Reliability at Scale

The larger the job, the more often something breaks during it. Meta's Llama 3 paper is the best public data point: during a 54-day snapshot of pre-training on up to 16K GPUs, there were 466 job interruptions, 419 of them unexpected. About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues, and GPU issues were the largest category at 58.7%. Meta still achieved more than 90% effective training time.[30] The lesson for a university: at a few hundred GPUs failures are occasional; at thousands they are daily, and your tooling and your users' checkpointing (see Training at Scale) must assume it.

Xid errors

The NVIDIA driver logs GPU errors to the kernel log as Xid events. NVIDIA publishes an Xid catalog giving each code's meaning and a recommended recovery action.[40] Some you will see regularly:

All rows from the NVIDIA Xid catalog.[40]

Operationally: alert on Xid events (dcgm-exporter exposes DCGM_FI_DEV_XID_ERRORS by default and an exporter-owned counter DCGM_EXP_XID_ERRORS_TOTAL)[9], classify them as user-caused (13, 31, 43) or hardware (48, 95, 140, link errors), and drain nodes automatically for the hardware class. Watching Xid 13 and 31 by user is also a good way to find people who need help debugging.

# Recent Xid events on a node
journalctl -k --since "24 hours ago" | grep -i xid

ECC and memory health

nvidia-smi reports volatile and aggregate ECC counts; single-bit errors are corrected by hardware, while double-bit errors are detected but not corrected.[2] Current NVIDIA GPUs handle failing HBM rows by row remapping, and dcgm-exporter's default metrics include correctable and uncorrectable remapped-row counts and a row-remap-failure flag.[9] On AMD, amd-smi bad-pages lists pending and retired pages, and amd-smi monitor --ecc reports single-bit, double-bit and PCIe replay counts.[5] A rising count on one GPU is a reason to schedule a replacement before it becomes a job-killing uncorrectable error.

Stragglers

A straggler is a component that has not failed but is slow: a GPU clocking down from a thermal problem, a NIC with a degraded link, a node with a bad DIMM. In synchronous training or tightly coupled MPI, everyone waits for the slowest rank. Meta observed that a single straggler "can slow down thousands of other GPUs", often appearing as functioning but slow communication, and also saw a diurnal 1-2% throughput variation driven by mid-day temperatures.[30] Detection techniques:

• Per-rank step timing logged by the application; the slowest rank is the suspect.
• Clock event reasons from nvidia-smi -q -d PERFORMANCE or nvidia-smi dmon -s v (power and thermal violations) to catch throttling GPUs.[2]
• Periodic nccl-tests across node groups, comparing bus bandwidth to the acceptance baseline.[22]
• UFM link-quality counters and link-down events on the fabric.[20]
• PyTorch's NCCL flight recorder, which Meta used to capture collective metadata and diagnose hangs and slow collectives.[30]

Field note: Keep a "suspect list" of nodes with any recent Xid, ECC, link or throttling event, and exclude them from large jobs (sbatch --exclude=) until a level 3 diagnostic passes. It is cheaper to idle one node than to have it slow down 500 others.

Key Takeaways

• Deploy DCGM and dcgm-exporter (or amd-smi based exporters) on every GPU node and label metrics with job IDs; node-level dashboards alone cannot answer efficiency questions.
• "GPU utilization" only means a kernel was running. Track SM activity, tensor-pipe activity and DRAM activity, and MFU for training.[2][6][29]
• Enable gres/gpu TRES in Slurm to get GPU hours plus gpuutil and gpumem per job, and feed it into Open XDMoD for center-level reporting.[11][14]
• Automate health: NHC for basics, dcgmi diag -r 1 after jobs, deeper levels after service, nccl-tests and UFM telemetry for the fabric.
• Profile in order: PyTorch profiler or Nsight Systems/rocprofv3 timeline first, Nsight Compute on the few kernels that matter, HPCToolkit or TAU for large MPI codes.
• Report energy per job and test power caps against time-to-solution; know that Slurm's GPU energy plugin currently supports rsmi only.[32]
• At thousands of GPUs failures are daily; alert on Xid and ECC events, hunt stragglers, and drain suspect nodes before large jobs land on them.[30][40]

Sources

1. PowerEdge: Useful NVIDIA-SMI queries for troubleshooting (Dell Technologies)
2. nvidia-smi documentation (NVIDIA)
3. Remove rocm-smi for ROCm 7.1, rocm-systems commit (AMD ROCm, GitHub)
4. Getting to Know Your GPU: A Deep Dive into AMD SMI (AMD ROCm Blogs, 2024)
5. AMD SMI CLI tool usage (AMD ROCm documentation)
6. DCGM Profiling module (NVIDIA DCGM documentation)
7. Install DCGM Exporter (NVIDIA DCGM documentation)
8. dcgm-exporter command reference (NVIDIA DCGM documentation)
9. dcgm-exporter default-counters.csv (NVIDIA, GitHub)
10. DCGM Exporter repository README (NVIDIA, GitHub)
11. Trackable RESources (TRES) (SchedMD Slurm documentation)
12. Generic Resource (GRES) Scheduling (SchedMD Slurm documentation)
13. sacct manual page (SchedMD Slurm documentation)
14. Open XDMoD: GPU Metrics (University at Buffalo CCR)
15. xdmod-supremm CHANGELOG (University at Buffalo CCR, GitHub)
16. Job Statistics with NVIDIA Data Center GPU Manager and SLURM (NVIDIA Technical Blog, 2019)
17. DCGM Feature Overview: job statistics (NVIDIA DCGM documentation)
18. LBNL Node Health Check (NHC) README (GitHub)
19. dcgmi diag (NVIDIA DCGM documentation)
20. UFM Telemetry, InfiniBand Cluster Bring-up Procedure (NVIDIA)
21. acct_gather.conf (SchedMD Slurm documentation)
22. nccl-tests README (NVIDIA, GitHub)
23. Nsight Systems User Guide (NVIDIA)
24. Nsight Compute CLI (NVIDIA)
25. Application tracing and profiling using rocprofv3 (AMD ROCm documentation)
26. torch.profiler (PyTorch documentation)
27. HPCToolkit: Monitoring GPU Operations (HPCToolkit documentation, 2025.1.2)
28. Tuning and Analysis Utilities (TAU) (OLCF User Documentation)
29. PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., arXiv, 2022)
30. The Llama 3 Herd of Models (Meta, arXiv, 2024)
31. LLM inference benchmarking metrics (NVIDIA NIM documentation)
32. slurm.conf manual page (SchedMD Slurm documentation)
33. The Linpack Benchmark (TOP500)
34. HPL-MxP Rules (hpl-mxp.org)
35. HPCG Benchmark (hpcg-benchmark.org)
36. MLPerf Training benchmark suite (MLCommons)
37. MLCommons Releases New MLPerf Inference v6.0 Benchmark Results (MLCommons, 2026)
38. MLPerf Training: HPC (MLCommons)
39. TOP500 June 2026 (TOP500, 2026)
40. Analyzing Xid Errors with the Xid Catalog (NVIDIA)