Systems Software and Provisioning

The 2018 version of this chapter was about PXE, TFTP, IPMI, ROCKS, Cobbler and NIS. PXE is still there, but CentOS, ROCKS and Warewulf 3 are gone, and the hard part of bringing up a node is no longer the OS image. It is the GPU driver, the NVSwitch fabric daemon, the RDMA stack, the CUDA or ROCm toolkits, and the modules, Python environments and containers that must line up with all of it. This chapter walks the stack from the baseboard management controller to the user's module load, with current versions as of September 2026 and the compatibility rules that decide whether a job runs or fails with a cryptic driver error. For the hardware underneath see Accelerators and Interconnects; for Slurm and Kubernetes on top see Scheduling and Orchestration.

What Changed Since 2018

A quick map from the old chapter to the current equivalents.

Choosing the Operating System

Enterprise Linux after CentOS

Many former CentOS sites moved to RHEL or one of the RHEL-compatible rebuilds, which track RHEL minor releases:

• RHEL itself. RHEL 8, 9 and 10 each have a ten-year life cycle across Full Support and Maintenance Support, followed by an Extended Life Phase; Red Hat's Extended Life Cycle Premium add-on now replaces the old EUS offerings for pinning to a minor release[5]. The current minor release is RHEL 10.2, which adds ML-KEM post-quantum key exchange to OpenSSH in FIPS mode[6].
• Rocky Linux. Rocky Linux 10.2 went GA on May 28, 2026, and Rocky Linux 9.8 is the current EL9 release[7][8]. In 10.2 the image builder gained a pxe-tar-xz output format for stateless PXE images for HPC and diskless systems, and the default /boot partition grew from 1 GiB to 2 GiB to fit larger initramfs images[7].
• AlmaLinux. AlmaLinux 10.2 ("Lavender Lion") ships kernel 6.12.0-211.7.3.el10_2[9]. Its most useful deviation from RHEL for clusters with older nodes: RHEL 10 raised the x86 baseline to x86-64-v3, and AlmaLinux also builds an x86-64-v2 architecture for older hardware[9]. AlmaLinux 10.3 and 9.9 were in beta in September 2026[10].

The EL10 transition is the live issue in 2026. Rocky 10.2 ships GCC 14.3 as the system compiler plus GCC Toolset 15 and Python 3.14[7], but the ecosystem is still catching up. OpenHPC 4.x targets EL10[11], Warewulf added EL10 support in 4.6.4[12], and xCAT's EL10 support is still described as "coming"[13].

Ubuntu LTS

Ubuntu is common on GPU systems, partly because vendor AI containers are Ubuntu-based (the NGC PyTorch 26.08 image is built on Ubuntu 24.04)[14]. Ubuntu 26.04 LTS ("Resolute Raccoon") was released April 23, 2026, with five years of standard security updates (to April 2031) and ten years with an Ubuntu Pro subscription[15][16]. It also ships authd, a service for authenticating against cloud identity providers, in the official repositories[15]. Ubuntu 24.04 LTS remains the conservative choice while vendor support for 26.04 fills in.

Let the Support Matrices Decide

Pick the newest OS release that appears on every support matrix you depend on. They do not agree:

Note the gaps: the ROCm 7.14 table lists Rocky Linux 9 but not 10, and RHEL 10 but not Ubuntu 26.04[18]. ROCm on Rocky 10 may work fine, but it is your problem when it does not.

Field note: Check the OS support table for the GPU driver, the ROCm or CUDA release, the network stack (DOCA-OFED), the parallel file system client and the cluster manager before you pick an OS. The file system client is usually the one that lags. The distribution with the most features is irrelevant if your Lustre or GPFS client will not build against its kernel.

Bare-Metal Provisioning

Out-of-Band Management: Redfish Replaces IPMI

IPMI was the old workhorse for remote power and console. Its replacement is DMTF's Redfish, which "uses RESTful interface semantics to access a schema-based data model to conduct management operations," and covers servers plus most other data center equipment, in-band and out-of-band[2]. The current specification is DSP0266 version 1.23.0 (September 2025)[2]. You can drive it with curl:

# Example only: BMC address, credentials and the system ID ("1") vary by vendor.
# List the systems managed by this BMC
curl -sk -u admin:"$BMC_PASS" https://bmc-node001.example.edu/redfish/v1/Systems

# Force a restart (the ResetType values a BMC accepts are listed in
# ResetType@Redfish.AllowableValues on the ComputerSystem resource)
curl -sk -u admin:"$BMC_PASS" -X POST \
  -H "Content-Type: application/json" \
  -d '{"ResetType": "ForceRestart"}' \
  https://bmc-node001.example.edu/redfish/v1/Systems/1/Actions/ComputerSystem.Reset

The URI structure and the reset action with its allowable values (On, ForceOff, ForceRestart and others) come straight from the specification's examples[2]. Redfish matters more on GPU racks than it did on CPU clusters: NVIDIA Base Command Manager 11 talks to the BMCs of GB200 compute and switch trays over Redfish to fetch metrics and detect leaks[19]. IPMI has not vanished, though; BCM 11.34.0 still added an IPMI plugin to Slurm[20].

Warewulf 4

Warewulf is a widely used open-source provisioner for stateless HPC nodes. Warewulf 4 was rewritten around two ideas: node images are built from OCI container images, and per-node configuration is rendered from templated overlays driven by node and profile settings. The current feature release is v4.7.0, which added server TLS support, a new wwctl unset command and refactored server routes; the 4.6.x series added EL10 support (4.6.4) and a preview of provisioning images to local disk instead of RAM (4.6.2)[12]. The Enterprise Linux quickstart shows how short the basic flow is[21]:

# From the Warewulf EL quickstart (addresses are examples)
wwctl configure --all
wwctl image import docker://ghcr.io/warewulf/warewulf-rockylinux:9 rockylinux-9 --build
wwctl profile set default --image rockylinux-9
wwctl profile set -y default --netmask=255.255.252.0 --gateway=10.0.0.1
wwctl node add n1 --ipaddr=10.0.2.1 --discoverable=true
wwctl overlay build

Because the image is a container image, you can build and test the GPU node image in a normal container workflow, then import it: the biggest operational improvement over Warewulf 3.

OpenHPC 4

OpenHPC remains the community "building blocks" repository: pre-built RPMs for provisioning, Slurm, compilers, MPI stacks and scientific libraries, plus install recipes. OpenHPC v4.0 (October 2025) targets EL 10 and openEuler 24.03, retired Warewulf 3 after more than ten years, and supports three provisioners: Confluent, OpenCHAMI and Warewulf 4[1]. Components are built with GCC 15.1, and 4.x is intended for fresh installs, not upgrades from 3.x[1]. The current release is 4.1, with 480 packages for EL 10[11]. The 3.x branch continues with release 3.5[22].

Confluent and the xCAT Reversal

xCAT, the other long-time open provisioner, was announced for end of life, then un-sunset. The current README says VersatusHPC has joined the xCAT Consortium, EL10 support is coming, updates will continue "while there is community and user demand," and the Consortium still recommends Confluent as the long-term successor[13]. Confluent describes itself as "the modern successor of xCAT," supports x86_64 and aarch64, and provisions both stateful and stateless images[23]. Existing xCAT sites are not forced off it, but new deployments should start elsewhere.

OpenCHAMI (Open Composable Heterogeneous Adaptable Management Infrastructure) is the newer, cloud-style option aimed at "cloud engineers and HPC sysadmins"[24]. It is worth a look for large sites that want microservice-style system management, and it is one of OpenHPC 4's supported provisioners[1].

NVIDIA Base Command Manager

Base Command Manager (BCM) is NVIDIA's commercial cluster manager and the path of least resistance for DGX and large NVIDIA systems. BCM 11 installs the head node OS and creates a default compute image from the same distribution; it is also included in NVIDIA Mission Control, which adds DGX GB200 rack features such as provisioning compute trays and switch trays[19]. BCM 11.34.0 (September 11, 2026) added Slurm 26.05, CUDA 13.2 packages, Ubuntu 26.04, Vera Rubin NVL72 bring-up, Pyxis 0.24.0 and Enroot 4.2.0, and cluster-wide SSH multi-factor authentication[20]. The trade-off: a license and lock-in for a tested, integrated stack.

MAAS and Metal3: Cloud-Style Bare Metal

Two tools come from the cloud side. Canonical's MAAS (Metal as a Service) makes "physical servers behave like cloud instances," with hardware discovery, API-driven OS deployment and integrated DHCP, DNS and image management[25]. Metal3 manages bare-metal hosts through Kubernetes, using OpenStack Ironic under a Bare Metal Operator and Cluster API integration[26]. Both fit best when GPU nodes run Kubernetes rather than Slurm (see Scheduling and Orchestration).

Stateless or Stateful?

Stateless (RAM-booted) images remain the right compute-node default. Two GPU-era complications: first, GPU node images are large (driver, DOCA-OFED, Fabric Manager) and sit in RAM that jobs could use. Warewulf 4.7.0 documents moving the image to swap to reclaim RAM[27], and 4.6.2 previewed provisioning to local disk[12]. Second, local NVMe on GPU nodes is valuable for scratch and checkpoint staging; do not let a diskless mindset waste it.

Field note: Treat the node image like software: definition in git, built in CI, booted on one canary node, burned in (NCCL tests, a short MPI job, GPU health checks), then rolled out. Keep the previous image so rollback is a profile change and a reboot.

Configuration Management

With stateless nodes, configuration management splits into three layers:

1. The image: packages, drivers, kernel modules. Changes require a rebuild and reboot. Build from a Containerfile or Warewulf/BCM image tooling.
2. Overlays or templates: per-node files rendered at boot (network config, hostnames, slurm.conf, SSSD config, SSH host keys and certificates). Warewulf's overlays and BCM's categories live here.
3. Runtime configuration: things that change without a reboot. Ansible is the de facto tool for this and for the stateful servers (head nodes, login nodes, storage and identity servers)[28].

Clusters get into trouble when all three layers edit the same file. Fold one-off Ansible changes back into the image.

The GPU Software Stack

The stack, from the bottom up:

Only the first four rows belong to the administrator. Toolkits, libraries and frameworks can coexist in many versions, provided each is compatible with the node's one driver.

The NVIDIA Kernel Driver

The NVIDIA Linux driver is a set of kernel modules: nvidia.ko, nvidia-modeset.ko, nvidia-uvm.ko, nvidia-drm.ko and nvidia-peermem.ko. They are now open source, dual licensed MIT/GPLv2, with source published for every release, and for releases after driver 570 the package names follow the "open" convention[17]. The installation guide shows the per-distribution commands, including how to pin a driver branch so a routine dnf update does not move you to a new one[17]:

# RHEL/Rocky/Alma 8 or 9: pin to the R580 branch with a DNF module stream
dnf module enable nvidia-driver:580-open
dnf install nvidia-open

# Ubuntu 22.04 / 24.04 / 26.04: pin to R580, then install
apt install nvidia-driver-pinning-580
apt install nvidia-open

Which branch should you pin? The CUDA release notes map each toolkit to a driver branch: CUDA 13.0 -> R580, 13.1 -> R590, 13.2 -> R595, 13.3 -> R610, 13.4 -> R615[29]. Also note a packaging change: starting with CUDA 13.4, the Linux CUDA toolkit no longer bundles the driver[29]. That formalizes what good clusters already did: install the driver in the node image and the toolkit separately.

CUDA Compatibility Rules

Most "CUDA doesn't work" tickets are compatibility problems. Teach these three rules to users as well as staff.

1. Backward compatibility. An application built with an older toolkit runs on a newer driver. Always true[29].
2. Minor version compatibility. Since CUDA 11, an application built with any toolkit in a major family runs on any driver at or above that family's minimum, "with limited feature-set"[30]. For CUDA 13.x the minimum is driver 580; for 12.x it is 525 (with drivers below 580); for 11.x, 450[29]. So a node on an R580 driver can run applications built with CUDA 13.4, but features new in 13.4 need an R615 or later driver[29]. Documented limits: device code compiled to PTX will not run on older drivers, and builds need an explicit target such as nvcc -arch=sm_xx[30].
3. Forward compatibility. To run an application built with a newer major CUDA version on an older driver branch, install the cuda-compat-<major>-<minor> package, which puts newer user-space driver libraries in /usr/local/cuda-X.Y/compat/. It does not configure anything; you must add that directory to LD_LIBRARY_PATH. And it works only on NVIDIA data center GPUs, select NGC Server Ready RTX SKUs and Jetson boards, not consumer GeForce cards[31].

# Forward compatibility example (data center GPUs only)
dnf install cuda-compat-12-8           # user-space driver libs from the R570 driver
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/compat:$LD_LIBRARY_PATH

The practical policy: run one recent production driver branch on all GPU nodes, upgrade it a few times a year, and let users choose toolkits freely within the major family.

Field note: The "CUDA Version" at the top right of nvidia-smi is the CUDA version that matches the installed driver, not the toolkit anyone is using; NVIDIA's own example shows a driver reporting CUDA 13.0 running a CUDA 13.4 application[30]. Users read it as "the cluster has CUDA 13.0" and then install the wrong wheels. Put a one-paragraph explanation in your user docs.

Fabric Manager for NVSwitch Systems

On HGX and DGX baseboards where the GPUs talk through NVSwitch chips, NVLink does not work until the nvidia-fabricmanager service has configured the switches. The package installs the systemd unit, but the administrator must enable and start it[32]. Two details bite in practice:

• Version lockstep. At startup Fabric Manager checks the loaded kernel driver stack and aborts if the versions are incompatible[32]. Install Fabric Manager from the same driver branch as the kernel driver, in the same image build, and pin both.
• Blackwell adds NVLSM. On DGX and HGX B200/B300 and HGX B100 and later, Fabric Manager depends on the NVLink Subnet Manager (NVLSM), derived from the InfiniBand subnet manager, which programs the NVSwitch routing tables. These systems also need an OFED driver plus the libibumad3 and infiniband-diags packages[32]. Minimum driver versions are 525 for HGX H100 and 570 for HGX B200/B300[32].

sudo systemctl enable nvidia-fabricmanager
sudo systemctl start nvidia-fabricmanager
sudo systemctl status nvidia-fabricmanager

On rack-scale NVLink systems such as DGX GB200, the NVLink switches sit in separate switch trays, which BCM with a Mission Control license provisions and manages alongside the compute trays[19]. See Interconnects for the NVLink topology itself.

GPUDirect RDMA, nvidia-peermem and DOCA-OFED

GPUDirect RDMA lets a NIC read and write GPU memory directly, which multi-node NCCL and CUDA-aware MPI need. Two mechanisms:

• nvidia-peermem, a kernel module shipped with the driver since CUDA 11.4, which gives NVIDIA InfiniBand HCAs peer access to GPU memory and replaced the old out-of-tree nv_peer_mem module[33]. Load it with modprobe nvidia-peermem. Ordering matters: if the GPU driver was installed before MLNX_OFED, reinstall the driver so nvidia-peermem is built against the OFED RDMA APIs[33].
• Linux dma-buf. NCCL has supported dma-buf registration since 2.13 through NCCL_DMABUF_ENABLE, enabled by default and turned off automatically if the kernel or drivers do not support it[34]. It is an alternative registration path to nvidia-peermem where the kernel and drivers support it.

On the network side, NVIDIA's host driver stack is now DOCA-OFED, described as "a 1-to-1 substitute for MLNX_OFED." The last standalone MLNX_OFED release was October 2024; it receives only critical bug and security fixes until October 2027, when it reaches end of life[4]. DOCA-Host is packaged in profiles (doca-all, doca-networking, doca-ofed, doca-roce), and the switch is an ordinary package install once the repository is configured: dnf -y install doca-ofed[4]. Plan the switch now.

Field note: Test GPUDirect RDMA explicitly on every new image. Run the NCCL all-reduce test across two nodes with NCCL_DEBUG=INFO and confirm the log shows GPU Direct RDMA in use. A missing module or wrong load order does not cause an error; NCCL quietly falls back to staging through host memory and your multi-node bandwidth drops. Repeat the test inside your recommended container: a missing RDMA provider library there causes the same silent fallback.

NCCL, cuDNN and the Library Layer

NCCL is NVIDIA's topology-aware collective communication library; the current GitHub release is 2.32.3[35]. cuDNN 9.26.0 adds support for CUDA 13.4 and for Rubin-architecture GPUs (compute capability 10.7)[36]. Neither needs a system-wide install: framework wheels and NGC containers bundle their own. The system's job is a new enough driver and a working RDMA path; where sites build software against these libraries, provide them as modules.

The ROCm Stack

AMD's ROCm has two release lines in 2026. The 7.2.x line continued with maintenance releases through 7.2.4 (May 29, 2026)[37]. ROCm Core SDK 7.14.0 (July 15, 2026) moved ROCm to "TheRock," a modular build and release system with a leaner core SDK plus optional domain-specific SDKs for AI, data science and HPC; its version number follows a gap from the 7.9 through 7.13 preview releases[38]. 7.14.1 (September 2, 2026) fixed a regression in which fault-injection instrumentation compiled into RCCL cut collective performance on InfiniBand for MI300X[38].

Supported data center GPUs in the current system requirements include the MI355X and MI350X (CDNA4, LLVM target gfx950), MI325X, MI300X and MI300A (CDNA3, gfx942) and the MI200 series (gfx90a)[18]. Spack's amdgpu_target variant and HIP compiler flags use these gfx targets. RCCL is AMD's collective library, implementing the same collectives as NCCL over PCIe, xGMI and networks[39], and frameworks call it through the same NCCL-style APIs.

User Software Stacks

Spack

Spack reached v1.0 on July 20, 2025, after almost seven years of work on compiler dependencies. It split the package repository from the tool and introduced a stable, separately versioned package API[40]. v1.2.0 (June 21, 2026) made a new parallel installer the default and added concretization groups, experimental build sandboxing and SBOM generation[41]. Spack's spec syntax expresses versions, variants, compilers, targets and dependencies in one line[42]:

# Get Spack
git clone --depth=2 https://github.com/spack/spack.git
. spack/share/spack/setup-env.sh

# One-off install with a full spec (example from the Spack docs)
spack install mpileaks@1.2:1.4 +debug ~qt target=x86_64_v3 %gcc@15 ^libelf@1.1 %clang@20

# Environments are the recommended way to manage a stack
spack env create myenv
spack env activate myenv
spack add curl @8 python py-numpy py-scipy py-matplotlib
spack concretize
spack install

The environment commands are from the Spack environments guide[43]. For GPU stacks, many packages have +cuda or +rocm variants, and you should usually point Spack at the system CUDA or ROCm installation instead of letting it download its own. The documented pattern is an external in packages.yaml[44]:

# packages.yaml (example values: set cuda_arch for your GPUs and the real prefix)
packages:
  all:
    variants:
      - cuda_arch=90
  cuda:
    buildable: false
    externals:
      - spec: cuda@13.0.2
        prefix: /usr/local/cuda-13.0

Spack generates Lmod (hierarchical) or Tcl module files; spack module lmod refresh regenerates them[45]. Note that Spack 1.0 made breaking changes to spec semantics (including the meaning of %) and to package repositories, with migration commands provided[40], so recipes and site configs written for 0.x need review.

EasyBuild

EasyBuild takes a different approach: curated recipes ("easyconfigs") built on a small set of versioned compiler toolchains such as foss and intel[46]. The current release, 5.4.0 (August 9, 2026), supports 3,091 software packages and 31 toolchains[47]. EasyBuild 5.0 enabled RPATH linking by default and requires Lmod 8.0 or newer when Lmod is the modules tool[48]. Installation with dependency resolution is one command[46]:

eb -S ^GCC-14                      # search available easyconfigs
eb HPL-2.3-foss-2021b.eb --robot   # build HPL and everything it needs (example from the docs)

Spack offers maximum flexibility; EasyBuild a curated stack organized by toolchain generation. Pick one; running both doubles the module tree users must learn.

Lmod

Lmod's documentation is at version 9.3[49]. Users need five commands: module avail, module spider (searches all modules, including those hidden behind a hierarchy), module load, module swap and module purge[49]. A typical GPU job preamble:

# Module names and versions are site-specific examples
module purge
module load gcc/14.3.0 cuda/13.0 openmpi/5.0
module list

Use Lmod's compiler -> MPI hierarchy: it stops users from mixing libraries built against different MPIs.

The Python Environment Problem

Python is where most AI users live and what staff control least: thousands of small files in home directories that hammer the metadata server, pip wheels that bundle their own CUDA libraries, and conda init editing everyone's .bashrc. In 2026 there are also licensing and tooling shifts.

Anaconda licensing. The conda tool is free and open source, but Anaconda's defaults channel packages require a paid license for organizations of 200 or more people not using them for educational purposes, and because conda points at defaults unless configured otherwise, users can incur license exposure without knowing it. LLNL is phasing out Anaconda and will block its paid channels site-wide in February 2027, recommending conda-forge with Miniforge as a drop-in replacement[3]. Read Anaconda's current terms yourself[50] and at minimum set conda-forge as the default channel in a system-wide .condarc.

uv. Astral's uv is a Rust-based Python package and project manager with a global cache for deduplication; it can create virtual environments with a specific Python version (uv venv --python 3.12)[51][52]. On clusters, the key detail is its cache: uv links files from the cache into environments, and if the cache is on a different file system than the environment it falls back to slow copies[53]. Set UV_CACHE_DIR to a directory on the same file system as the environments (for example project space rather than a small home quota).

pixi. prefix.dev's pixi manages conda-ecosystem packages per project with a manifest, lock file and multiple environments[54], a good fit when a project needs non-Python binaries that pip cannot provide.

# uv: per-project venv on project storage, cache on the same file system (paths are examples)
export UV_CACHE_DIR=/project/mylab/.cache/uv
cd /project/mylab/myproject
uv venv --python 3.12
source .venv/bin/activate
uv pip install torch numpy

Field note: Publish one recommended Python workflow: where environments live (not home), which tool (uv for pure Python, pixi or Miniforge when binaries are needed, a container for large frameworks) and how to load CUDA for extensions. Users follow a clear default; they do not read tool comparisons.

Containers for HPC and AI

For large AI frameworks, containers have won: a PyTorch stack with matching CUDA, cuDNN and NCCL is easier to consume as a vendor image than to rebuild. For MPI codes, containers need care at the MPI and network boundary. HPC runtimes differ from Docker in one key way: containers run as the user, not through a root daemon.

Apptainer

Apptainer, the continuation of Singularity, is the familiar HPC runtime at most centers; version 1.5.0 was released May 6, 2026[55]. It runs single-file SIF images and pulls directly from OCI registries. The --nv flag exposes /dev/nvidia* devices and binds the host's CUDA driver libraries into the container; --rocm does the same for AMD GPUs; and Apptainer also supports the Container Device Interface (CDI) and an experimental --nvccli mode using NVIDIA's container CLI[56].

# Pull an NGC PyTorch image into a SIF file (tag is an example; see the NGC catalog)
apptainer pull pytorch-26.08.sif docker://nvcr.io/nvidia/pytorch:26.08-py3

# Run with NVIDIA GPUs
apptainer exec --nv pytorch-26.08.sif python -c 'import torch; print(torch.cuda.is_available())'

# AMD GPUs use --rocm instead of --nv
apptainer exec --rocm rocm-pytorch.sif python -c 'import torch; print(torch.cuda.is_available())'

The pull syntax follows the Apptainer command reference[57]. Because --nv binds the host's user-space driver into the container, the CUDA compatibility rules above apply between the container's toolkit and the node's driver. The NGC PyTorch 26.08 container, for instance, is Ubuntu 24.04-based and built on CUDA 13.4.1[14], so under minor version compatibility it needs at least an R580 driver, with R615 needed for 13.4-specific features[29]. Host libraries are bound in, so the container's libc must not be too far from the host's[56].

Enroot and Pyxis

NVIDIA's Enroot is "an enhanced unprivileged chroot" with no daemon, no setuid binary and deliberately little isolation[58]. Pyxis is a Slurm SPANK plugin that adds container options directly to srun, pulling Docker images with layer caching and supporting multi-node MPI through PMI2 or PMIx[59]. NVIDIA's own cluster manager ships both (BCM 11.34.0 updated them to Pyxis 0.24.0 and Enroot 4.2.0)[20], and many AI users expect this workflow.

# Run a command inside a container on a compute node
srun --container-image=nvcr.io\#nvidia/pytorch:26.08-py3 \
     --container-mounts=/scratch/$USER:/data \
     python -c 'import torch; print(torch.__version__)'

# Multi-node job: PMIx wires up the ranks, Pyxis supplies the container (example sizes)
srun -N 2 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
     --container-image=nvcr.io\#nvidia/pytorch:26.08-py3 \
     --container-name=pt2608 \
     python train.py

In the image URI, # separates the registry from the image; it must be escaped in #SBATCH --container-image lines, where # starts a comment, and escaping it on the command line as above is harmless[59]. --container-name keeps the unpacked container on the node so later steps skip the import[59]. For admins, one gotcha: since Slurm 21.08, Pyxis must be compiled against the exact Slurm release you deploy, or Slurm refuses to load it with "Incompatible plugin version"[59]. Rebuild Pyxis as part of every Slurm upgrade.

Podman, Charliecloud and CDI

Podman in rootless mode is the closest thing to Docker that is safe on a shared cluster. NERSC worked with Red Hat to adapt it for HPC as podman-hpc, which lets Perlmutter users build images on the system and run them at scale, with support for Cray MPI and CUDA[60]. Charliecloud is a "lightweight, fully unprivileged" implementation that uses user namespaces and can build images, push and pull to registries, and use GPUs[61]. Underneath many of these, the Container Device Interface is replacing runtime-specific GPU hooks: since v1.12.0 the NVIDIA Container Toolkit can generate CDI specifications (nvidia-ctk cdi generate) that any CDI-aware runtime can consume[62].

Vendor Images: NGC and ROCm

NVIDIA's NGC catalog publishes framework containers on a regular cadence (tags like 26.08-py3) with release notes listing the exact CUDA, PyTorch and library versions[14]. AMD publishes validated rocm/pytorch images on Docker Hub with tags that encode ROCm, OS, Python and PyTorch versions, for example rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.9.1[63]. Use full tags or digests, never latest.

MPI, GPUs and the Network in Containers

Multi-node container jobs have to reach the host's network and cooperate with Slurm's process launch. Apptainer documents two models[64]:

• Hybrid model: MPI is installed in the container and launched by the host's Slurm or mpirun. It is simple, but the MPI in the container must be compatible with the host's, and must be configured for the host's network to perform well.
• Bind model: the container has no MPI; the host's MPI is bind-mounted in. This guarantees the tuned host MPI is used, but requires ABI compatibility and permission to bind host paths.

With Slurm, launch through srun --mpi=pmix (native in Open MPI since 3.1) so Slurm starts the ranks[65]. NCCL jobs also need the RDMA user-space libraries in the container and, in the cloud, the right NCCL network plugin. The NGC PyTorch 26.08 image, for example, now includes Google Cloud's NCCL plugins and activates them automatically on supported GCP machine types[14]. See Cloud and Hybrid for the cloud side.

Identity and Access

FreeIPA, SSSD and Kerberos

FreeIPA combines the 389 Directory Server, MIT Kerberos, DNS, the Dogtag certificate system and SSSD into one managed service. It defines Kerberos authentication and authorization policies, controls services such as DNS, sudo, SELinux and autofs, and can establish trusts with Active Directory[66]. That AD trust is the common campus pattern: users keep their campus AD credentials, while cluster-specific groups, POSIX attributes and sudo policy live in FreeIPA. SSSD is the client side on every node, enrolling machines into AD, FreeIPA or LDAP domains and caching identities and credentials[67]. Kerberos remains the SSO backbone inside the realm and underpins Kerberized NFS.

Federated SSO and CILogon

Research collaborations span institutions, and nobody wants to issue local accounts to every collaborator by hand. CILogon provides an identity and access management platform for research, combining federated identity (InCommon and eduGAIN) with COmanage collaborative organization management and OpenID Connect for applications[68]. Typical uses are OIDC login for web portals and issuing short-lived SSH credentials after federated login. BCM 11.34.0's cluster-wide SSH MFA setup[20] and Ubuntu 26.04's authd for cloud identity providers[15] show that MFA and IdP-backed login are now expected features, not add-ons.

SSH Certificates

SSH certificates replace sprawling authorized_keys and known_hosts files with a CA that signs user keys (with principals and a validity period) and host keys. Servers trust the CA via TrustedUserCAKeys, present signed host keys via HostCertificate, and can map principals with AuthorizedPrincipalsFile[69]. The OpenSSH tooling is built in[70]:

# Sign a user key valid for 12 hours for principal "jdoe" (names are examples)
ssh-keygen -s user_ca -I jdoe@example.edu -n jdoe -V +12h id_ed25519.pub

# Sign a host key for a login node
ssh-keygen -s host_ca -I login01 -h -n login01.hpc.example.edu ssh_host_ed25519_key.pub

# sshd_config on cluster nodes
#   TrustedUserCAKeys /etc/ssh/user_ca.pub
#   HostCertificate   /etc/ssh/ssh_host_ed25519_key-cert.pub

Short-lived user certificates give you revocation by expiry, and host certificates end the "host key changed" warnings after every reprovision of a stateless node.

Security Hardening Basics

NIST has now published two HPC-specific documents: SP 800-223 (final February 2024) defines an HPC system reference architecture, threat analysis and security posture[71], and SP 800-234 (final May 2026) is an HPC security overlay built on the moderate control baseline[72]. Start there for regulated data (see Research Computing Operations). The basics everywhere:

• Patch kernels promptly, GPU nodes included. For example, GhostLock (CVE-2026-43499), a use-after-free that lets an unprivileged local user become root and escape containers, affected every supported AlmaLinux release when disclosed in July[10]. Unprivileged containers do not protect against kernel bugs. Rebuild the GPU driver against the new kernel in the same image build.
• Isolate the management plane. BMCs, provisioning services and switch management belong on networks users cannot reach. Warewulf 4.7.0 added TLS for its server[12]; use it.
• Mind Secure Boot certificate transitions. Microsoft's 2011 UEFI Secure Boot certificates have expired; AlmaLinux 10.2's x86_64 shim is dual-signed with both the 2011 and 2023 certificates[9][10]. With Secure Boot on, test new shims and firmware on a canary node and sign any out-of-tree kernel modules.
• Watch crypto policy changes. On Rocky Linux 10.2 the FUTURE system-wide crypto policy now allows only hybrid ML-KEM key exchange, which will break connections to endpoints that do not support post-quantum algorithms[7]. Test policy changes against license servers, storage appliances and BMCs first.
• Keep SELinux or AppArmor on where you can. Rocky 10.2's SELinux policy now confines the redfish-finder service[7]; distribution policies increasingly cover HPC-adjacent services.
• Track the supply chain. Pin image digests, prefer signed images, and try Spack 1.2 SBOM generation[41].

Key Takeaways

• Pick the OS from the intersection of your vendors' support matrices. In 2026 that is usually Rocky/Alma/RHEL 9 or 10, or Ubuntu 24.04/26.04, and not every matrix includes EL10 or 26.04 yet.
• Warewulf 4, Confluent and OpenCHAMI are the open provisioners; BCM is the commercial path for large NVIDIA systems. Drive BMCs through Redfish.
• Build node images like software: container-based definitions in git, a canary node, a GPU and fabric burn-in, and an easy rollback.
• Install the NVIDIA driver, Fabric Manager (plus NVLSM on Blackwell HGX), DOCA-OFED and nvidia-peermem together in the image and pin the driver branch. Let users choose CUDA toolkits within the major family through minor version compatibility.
• MLNX_OFED reaches end of life in October 2027; move to DOCA-OFED now. ROCm moved to the modular TheRock-based Core SDK with 7.14.
• Use Spack or EasyBuild with Lmod for compiled software, and publish one clear Python workflow (conda-forge rather than Anaconda defaults, uv or pixi, environments off home directories).
• Deliver AI frameworks as containers (Apptainer, or Enroot + Pyxis with Slurm) and verify multi-node NCCL bandwidth inside the container, not just on bare metal.
• Centralize identity (FreeIPA or AD via SSSD), federate external users (CILogon, OIDC), and move to short-lived SSH certificates with MFA.

Sources

1. OpenHPC v4.0 Release: Major OS Support and New Provisioning Systems (OpenHPC, 2025)
2. Redfish Specification DSP0266 version 1.23.0 (DMTF, 2025)
3. Technical Bulletin 602: LLNL is phasing out support for Anaconda (LLNL, 2026)
4. DOCA Profiles and MLNX_OFED to DOCA-OFED Transition Guide (NVIDIA, 2026)
5. Red Hat Enterprise Linux Life Cycle (Red Hat, 2026)
6. Overview of Red Hat Enterprise Linux 10.2, 10.2 Release Notes (Red Hat, 2026)
7. Rocky Linux 10.2 Available Now (Rocky Linux, 2026)
8. Rocky Linux 9.8 Available Now (Rocky Linux, 2026)
9. AlmaLinux 10.2 Release Notes (AlmaLinux, 2026)
10. AlmaLinux Blog: 10.3 and 9.9 betas, Secure Boot certificates, GhostLock (AlmaLinux, 2026)
11. OpenHPC 4.x wiki: Community building blocks for HPC systems (4.1) (OpenHPC, 2026)
12. Warewulf releases v4.6.2 through v4.7.0 (Warewulf, 2025-2026)
13. xCAT README: The xCAT sunset was only a quick eclipse (xCAT Consortium, 2026)
14. PyTorch Release 26.08, NVIDIA Frameworks Release Notes (NVIDIA, 2026)
15. Canonical releases Ubuntu 26.04 LTS Resolute Raccoon (Canonical, 2026)
16. Ubuntu 26.04 LTS release notes (Canonical, 2026)
17. NVIDIA Driver Installation Guide: Kernel Modules (NVIDIA, 2026)
18. ROCm System requirements (Linux), ROCm 7.14 (AMD, 2026)
19. Base Command Manager 11 Release Notes: Overview (NVIDIA, 2025)
20. Release notes for NVIDIA Base Command Manager 11.34.0 (NVIDIA, 2026)
21. Warewulf Enterprise Linux Quickstart (Warewulf, 2026)
22. OpenHPC 3.x wiki (3.5) (OpenHPC, 2026)
23. Confluent documentation (xCAT Consortium, 2026)
24. OpenCHAMI: Secure, Scalable HPC System Management (OpenCHAMI, 2026)
25. MAAS documentation (Canonical, 2026)
26. Metal3: bare metal host management with Kubernetes (Metal3 project, 2026)
27. Warewulf v4.7.0 Release Notes (Warewulf, 2026)
28. Getting started with Ansible (Ansible Community Documentation, 2026)
29. CUDA Toolkit 13.4 Update 1 Release Notes (NVIDIA, 2026)
30. CUDA Compatibility: Minor Version Compatibility (NVIDIA, 2026)
31. CUDA Compatibility: Forward Compatibility (NVIDIA, 2026)
32. NVIDIA Fabric Manager User Guide (NVIDIA, 2025)
33. Developing a Linux Kernel Module using GPUDirect RDMA (NVIDIA, 2026)
34. NCCL User Guide: Environment Variables (NVIDIA, 2026)
35. NCCL releases (NVIDIA, GitHub, 2026)
36. cuDNN Backend Release Notes, 9.26.0 (NVIDIA, 2026)
37. ROCm release history (AMD, 2026)
38. ROCm Core SDK 7.14.1 and 7.14.0 release notes (AMD, 2026)
39. RCCL: ROCm Communication Collectives Library (AMD, GitHub, 2026)
40. Spack v1.0.0 release notes (Spack, 2025)
41. Spack CHANGELOG, v1.2.0 (Spack, 2026)
42. Spack documentation: Spec Syntax (Spack, 2026)
43. Spack documentation: Environments basics (Spack, 2026)
44. Spack documentation: Using External GPU Support (Spack, 2026)
45. Spack documentation: Modules (Spack, 2026)
46. Using EasyBuild (EasyBuild, 2026)
47. EasyBuild release notes, v5.4.0 (EasyBuild, 2026)
48. EasyBuild v5.0 overview of changes (EasyBuild, 2025)
49. User Guide for Lmod, Lmod 9.3 documentation (Lmod project, 2026)
50. Anaconda Terms of Service (Anaconda, 2025)
51. uv documentation (Astral, 2026)
52. uv: Using Python environments (Astral, 2026)
53. uv: Caching (Astral, 2026)
54. Basic usage of Pixi (prefix.dev, 2026)
55. Apptainer News: Announcing Apptainer release 1.5.0 (Apptainer, 2026)
56. Apptainer User Guide: GPU and other Device Support (Apptainer, 2026)
57. Apptainer User Guide: apptainer pull (Apptainer, 2026)
58. Enroot (NVIDIA, GitHub, 2026)
59. Pyxis: container plugin for Slurm (NVIDIA, GitHub, 2026)
60. Using Podman-hpc at NERSC (NERSC, 2026)
61. Charliecloud Overview (Charliecloud, 2026)
62. NVIDIA Container Toolkit: Support for Container Device Interface (NVIDIA, 2026)
63. PyTorch on ROCm installation and Docker images (AMD, 2026)
64. Apptainer and MPI applications (Apptainer, 2026)
65. Slurm MPI Users Guide (SchedMD, 2026)
66. FreeIPA - Identity, Policy, Audit (FreeIPA project, 2026)
67. SSSD - System Security Services Daemon (SSSD project, 2026)
68. CILogon: An Integrated Identity and Access Management Platform for Science (CILogon, 2026)
69. sshd_config(5) manual page (OpenBSD/OpenSSH, 2026)
70. ssh-keygen(1) manual page (OpenBSD/OpenSSH, 2026)
71. NIST SP 800-223, High-Performance Computing Security: Architecture, Threat Analysis, and Security Posture (NIST, 2024)
72. NIST SP 800-234, High-Performance Computing (HPC) Security Overlay (NIST, 2026)