Scheduling and Orchestration

The scheduler is where policy meets hardware. A GPU cluster with a bad queue design strands expensive accelerators behind CPU-only jobs, lets one lab camp on every H100 for a week, or scatters a 32-node training run across switches that were never meant to carry its all-reduce traffic. This chapter covers the scheduling layer for a converged AI and HPC system as it stands in 2026: Slurm in depth (because it still runs most research clusters), Kubernetes and its batch add-ons, how the two now overlap, GPU sharing, web portals, workflow tools, and how to design queues for a population that mixes week-long simulations with notebooks and hyperparameter sweeps. Hardware and fabric details live in Accelerators and Interconnects; framework-level parallelism lives in Training at Scale.

The Scheduling Landscape in 2026

Two families of scheduler matter for research computing. Slurm is the batch workload manager of classic HPC: jobs with a known size and time limit, strict policy, a fixed pool of nodes, and first-class support for multi-node MPI and topology. Kubernetes is the container orchestrator of cloud-native computing: long-running services, loose resource requests, elastic pools, and a huge ecosystem of operators. SchedMD's own Slinky documentation puts the distinction plainly: Kubernetes excels at indefinite workloads with vague requirements on a single node with loose policy, while Slurm excels at finite workloads with well defined requirements and topology across multiple nodes with strict policy.[1]

Three facts shape the current picture:

• Slurm 26.05 is the current release. SchedMD announced it in May 2026.[2] Headline changes include new topology/ring and topology/torus3d plugins, an srun --async step mode for queuing large numbers of steps, dynamic memory resizing of running jobs, REST API support for managing partitions, and GPU allocation statistics in the OpenMetrics (Prometheus) endpoints.[3]
• NVIDIA now owns SchedMD. NVIDIA announced the acquisition in December 2025 and stated it will continue to develop and distribute Slurm as open-source, vendor-neutral software. The same announcement notes Slurm is used on more than half of the top 10 and top 100 systems on the TOP500 list.[4] AMD (RSMI) and Intel (oneAPI) GPU detection remain in the 26.05 GRES documentation.[5]
• The two worlds are converging. SchedMD's Slinky project runs Slurm clusters on Kubernetes and can also use Slurm as a Kubernetes scheduler,[6] while Kubernetes batch projects (Kueue, Volcano) have added quotas, gang scheduling and topology-aware placement that look a lot like HPC schedulers.[7][8]

Slurm for GPU Clusters

Slurm treats GPUs as generic resources (GRES). Everything GPU-specific (requests, binding, accounting, sharing) builds on that abstraction, and all of the --gpu* options require the select/cons_tres plugin.[9]

GRES Configuration

Each GPU node gets a gres.conf describing its devices, and slurm.conf declares the GRES counts per node. Let Slurm discover GPUs rather than hand-writing device files: AutoDetect supports nvml (NVIDIA, via the NVIDIA Management Library), rsmi (AMD), oneapi (Intel), nrt, and nvidia, and can be set globally or per node.[5]

The excerpt below is a starting point for a cluster with an 8-GPU training pool and a smaller shared pool. Node names, core counts, memory and GPU type labels are examples; take real values from slurmd -C on each node.

# slurm.conf (excerpt, example values)
SelectType=select/cons_tres
GresTypes=gpu,shard
AccountingStorageTRES=gres/gpu,gres/gpu:h100,gres/gpu:l40s,gres/shard
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
PrologFlags=Contain
JobAcctGatherType=jobacct_gather/cgroup

NodeName=gpu[001-032] Sockets=2 CoresPerSocket=56 ThreadsPerCore=1 RealMemory=1900000 Gres=gpu:h100:8
NodeName=gsh[01-04]   Sockets=2 CoresPerSocket=32 ThreadsPerCore=1 RealMemory=500000  Gres=gpu:l40s:4,shard:64

# gres.conf (on every GPU node)
AutoDetect=nvml

Two accounting details trip people up. First, when you track a typed GRES such as gres/gpu:h100, also track the generic gres/gpu, or requests like --gres=gpu:1 will not be accounted. Second, tracking gres/gpu also enables gres/gpumem and gres/gpuutil.[10]

Requesting GPUs: --gres, --gpus and Friends

Slurm offers several ways to ask for GPUs, and they are not interchangeable:[11]

Companion options size the rest of the job around the GPUs: --cpus-per-gpu (not compatible with --cpus-per-task), --mem-per-gpu (mutually exclusive with --mem and --mem-per-cpu), and --gpu-freq for clock settings.[11] Administrators can set partition defaults with DefCpuPerGPU and DefMemPerGPU so that a bare --gpus=1 still gets a sensible slice of the node.[12] One rule applies across all of them: typed and untyped requests must be consistent within a job. If the batch job asked for --gres=gpu:2, a step cannot then ask for --gres=gpu:tesla:2.[9]

GPU Binding with --gpu-bind

Allocation decides which GPUs a job owns; binding decides which task sees which GPU. --gpu-bind is shorthand for --tres-bind=gres/gpu:..., and the supported types are:[13]

• closest: bind each task to the GPU(s) closest to its CPUs (in a NUMA domain a task may get several).
• single:N: like closest, but each task gets exactly one GPU, with N tasks per GPU in block order.
• per_task:N: each task gets N GPUs, preferring ones with affinity to its cores. This is what --gpus-per-task sets implicitly.
• map:list and mask:list: explicit per-task GPU IDs or hex masks, with *count repetition (e.g. map:0*4,1*4).
• none: turn off implicit binding.

Add verbose, in front of any type to log the binding. When task/cgroup is active with ConstrainDevices=yes, map and mask IDs are relative to the GPUs allocated to the job (the first allocated GPU is 0 even if its global index is 3).[13] For PyTorch launched by torchrun, you usually do not bind at the Slurm level at all: one Slurm task per node launches one worker per GPU, and each worker selects its device from LOCAL_RANK. For MPI codes with one rank per GPU, --gpus-per-task=1 gives correct binding without extra flags.

cgroups and Isolation

Without device constraints, a job allocated one GPU can open all eight. Enforce isolation with cgroups:[14]

# cgroup.conf
CgroupPlugin=cgroup/v2
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes

ConstrainDevices=yes restricts the job's allowed devices to its GRES allocation (the default is no). The cgroup v1 plugin is deprecated and gets no new features, so new builds should use v2. SchedMD's example pairs this with ProctrackType=proctrack/cgroup, TaskPlugin=task/cgroup,task/affinity and PrologFlags=Contain in slurm.conf, and recommends pam_slurm_adopt so users cannot escape their cgroup by SSHing to a node.[14] Slurm 26.05 keys cgroup v2 directories off the new SLUID job identifier rather than the job ID, which matters if you have prolog, epilog or monitoring scripts that walk the cgroup tree.[3]

Field note: The first week after turning on ConstrainDevices, you will find the users whose scripts hard-code CUDA_VISIBLE_DEVICES=0,1,2,3 or call cuda:3 directly. Tell people in advance to trust the environment Slurm sets, and to print nvidia-smi -L at the top of every job so the log shows what the job actually received.

Topology-Aware Scheduling

Without a topology plugin, Slurm treats nodes as a one-dimensional list and places jobs by best fit.[15] That is fine for single-node jobs and hopeless for multi-node training, where the all-reduce pattern punishes a job that straddles oversubscribed uplinks. Slurm 26.05 documents four topology models.

Tree Topology

TopologyPlugin=topology/tree describes a switch hierarchy in topology.conf. Slurm finds the lowest switch that can satisfy the request and best-fits nodes beneath it. Users can add --switches=count@max-time to say "I want at most this many leaf switches, and I will wait up to this long for them." The admin caps that wait with SchedulerParameters=max_switch_wait. SchedMD's advice is to list leaf switches with their nodes plus one top-level switch rather than every link, because full detail slows scheduling for little gain. For InfiniBand, the documentation points to the community slurmibtopology tool to generate the file.[15] Note that node Weight settings override topology-based selection under the tree plugin.[15]

Block Topology

topology/block fits systems built from fixed, strongly connected units, such as groups of nodes behind one leaf switch or rack-scale NVLink domains (see Interconnects). You define base blocks of nodes, and BlockSizes lists the enforced block sizes at each level. Each block must hold at least the base size, and each successive size must be a power of two larger than the one before. Unlike tree, which grabs the first resources that fit, block placement is designed to reduce fragmentation.[16][15]

# topology.conf for topology/block (example: four 8-node base blocks)
BlockName=b1 Nodes=gpu[001-008]
BlockName=b2 Nodes=gpu[009-016]
BlockName=b3 Nodes=gpu[017-024]
BlockName=b4 Nodes=gpu[025-032]
BlockSizes=8,16,32

Users shape placement with --segment=N, which splits the allocation into N-node segments (the node count must be evenly divisible by it), plus two block-only flags: --spread-segments and --consolidate-segments.[15] There are documented limitations. Node-count ranges force the scheduler to evaluate every value. Using --nodelist can make a job unplaceable because topology takes precedence (use --exclude instead). You cannot ask for non-adjacent blocks.[15] --exclusive=topo stops other jobs from sharing the job's topology segment,[11] and 26.05 adds a matching partition setting, Exclusive=TOPO.[3]

Ring, Torus and Multiple Topologies

Slurm 26.05 added topology/ring (ordered rings of at most 16 nodes, with contiguous segments that may wrap) and topology/torus3d (contiguous sub-cube placements, configured only in topology.yaml).[15] Slurm 25.05 introduced topology.yaml, which lets you define several topologies and assign one per partition. This helps a converged cluster whose GPU pool uses block topology while its CPU pool uses a tree.[15] Hostlist functions such as block{b1} and switchwith{node0} work in commands and configs, for example scontrol update node=block{b1} state=resume.[15]

Field note: Topology plugins only help if the file matches the cabling. Regenerate topology.conf after every fabric change and diff it in version control. A stale topology file is worse than none, because the scheduler will confidently pack jobs onto what it thinks is one switch.

Priority, Fairshare, QOS and Preemption

Multifactor Priority and Fair Tree

With PriorityType=priority/multifactor, a job's priority is a weighted sum of age, association, fairshare, job size, partition, QOS and per-TRES factors, minus nice.[17] Since Slurm 19.05 the fairshare factor uses the Fair Tree algorithm by default. Under Fair Tree, if account A has a higher fairshare than its sibling B, every user under A ranks above every user under B, and an account coordinator cannot accidentally hurt their users relative to other accounts.[18] Past usage decays with PriorityDecayHalfLife, which defaults to 7 days.[12]

On a GPU cluster, the key setting is what counts as usage. By default jobs are billed only by allocated CPUs (TRESBillingWeights="CPU=1"). A job that takes eight GPUs and four cores would then barely dent its owner's fairshare. Set partition billing weights so GPUs dominate. SchedMD's example is TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0".[10] Choose your own GPU weight so that one GPU costs roughly what the CPUs and memory it strands would have cost.

Quality of Service

A QOS bundles priority, limits and preemption rights. QOS priority only counts if PriorityWeightQOS is non-zero, and QOS preemption only applies under PreemptType=preempt/qos.[19] Useful options include GrpTRES, MaxTRESPerUser, MaxTRESPerJob and MaxWall limits; UsageFactor (e.g. 0 for free scavenger time); and flags such as DenyOnLimit, NoReserve (for preemptible QOSes, so they do not block backfill), Relative (limits as percentages, since 23.11), RequiresReservation and NoDecay.[19][20] A QOS attached to a partition in slurm.conf can implement a "floating" partition that spans all nodes but is capped by GrpTRES.[19]

# Example QOS tiers for a GPU partition (values are examples)
sacctmgr add qos gpu-normal
sacctmgr modify qos gpu-normal set Priority=100 MaxTRESPerUser=gres/gpu=16 MaxWall=2-00:00:00

sacctmgr add qos gpu-scavenger
sacctmgr modify qos gpu-scavenger set Priority=0 UsageFactor=0 Flags=NoReserve GraceTime=120

# gpu-normal jobs may preempt gpu-scavenger jobs
sacctmgr modify qos gpu-normal set Preempt=gpu-scavenger

# Large runs only inside an admin-created reservation
sacctmgr add qos gpu-hero
sacctmgr modify qos gpu-hero set Flags=RequiresReservation MaxTRESPerJob=gres/gpu=256

Slurm 26.05 also accepts += and -= in TRES assignments (e.g. GrpTRES=gres/gpu+=2), which makes incremental quota changes less error-prone.[20] Note that QOS Priority has nothing to do with preemption: only the Preempt list decides who can preempt whom.[20]

Preemption

PreemptType is either preempt/partition_prio (partitions with a higher PriorityTier preempt lower ones) or preempt/qos. PreemptMode may be OFF, CANCEL, REQUEUE or SUSPEND, optionally with GANG, and WITHIN lets jobs in the same QOS preempt each other.[21] When a job is selected, Slurm sets its end time to now plus GraceTime and immediately sends SIGCONT and SIGTERM. At the new end time it sends SIGCONT, SIGTERM and SIGKILL.[21] PreemptExemptTime guarantees a minimum run time before a job can be chosen.[12]

On GPU partitions, REQUEUE (or CANCEL) is the practical mode. SUSPEND stops processes but leaves them resident on the node, so the GPU's memory stays occupied, and Slurm notes that licenses are not freed by suspension either.[21] Preemptible work only pays off if jobs checkpoint when they get SIGTERM. A minimal pattern:

#!/bin/bash
#SBATCH --qos=gpu-scavenger
#SBATCH --requeue
#SBATCH --signal=USR1@300
#SBATCH --gpus=1
#SBATCH --time=12:00:00

# train.py must: resume from the newest checkpoint on start, and
# save a checkpoint and exit on SIGTERM (preemption) or SIGUSR1 (near walltime).
srun python train.py --ckpt-dir "/scratch/$USER/ckpt/$SLURM_JOB_NAME" --resume latest

import signal

stop_requested = False

def _request_stop(signum, frame):
    global stop_requested
    stop_requested = True

signal.signal(signal.SIGTERM, _request_stop)   # sent on preemption
signal.signal(signal.SIGUSR1, _request_stop)   # sent by --signal=USR1@300

for step in range(start_step, total_steps):
    train_step()
    if stop_requested:
        save_checkpoint(step)
        break

--signal may fire up to 60 seconds earlier than requested, and --requeue makes the batch job eligible for requeue after preemption or node failure.[11] Checkpoint formats and cadence are covered in Training at Scale.

Reservations

Advanced reservations set aside nodes for maintenance, courses, hero runs or acceptance testing. The canonical maintenance example is:[22]

scontrol create reservation starttime=2026-10-06T08:00:00 duration=480 \
    user=root flags=maint,ignore_jobs nodes=ALL

Other flags include DAILY and WEEKLY for recurring class slots, FLEX to let jobs spill outside the reserved nodes, MAGNETIC to pull in eligible jobs without an explicit --reservation, and PART_NODES and ANY_NODES.[23] As a maintenance reservation approaches, Slurm starts only jobs whose time limits end before it. Accurate --time values therefore keep the cluster busy right up to the outage.[22]

Job Arrays and Heterogeneous Jobs

Job arrays submit many near-identical batch jobs with one command. --array=0-15%4 runs at most four tasks at once, each task sees SLURM_ARRAY_TASK_ID, and output patterns use %A (array job ID) and %a (task ID). The largest index is bounded by MaxArraySize minus one.[24] Arrays are the right tool for hyperparameter sweeps, ensemble simulations and per-sample inference, and the % throttle is how a good citizen avoids flooding a shared GPU partition.

Heterogeneous jobs allocate differently shaped components that start together, for example a CPU simulation feeding a GPU training or inference process. In a batch script, components are separated by #SBATCH hetjob, the script itself runs on the first node of the first component, and steps target components with srun --het-group=N.[25][13] Environment variables are exposed per component with a _HET_GROUP_N suffix (e.g. SLURM_JOB_NODELIST_HET_GROUP_1).[25] For preemption, all components must be eligible, and a single PreemptMode is applied to the whole heterogeneous job.[21]

#!/bin/bash
#SBATCH --job-name=sim-plus-surrogate
#SBATCH --partition=cpu --nodes=4 --ntasks-per-node=64 --time=06:00:00
#SBATCH hetjob
#SBATCH --partition=gpu --nodes=1 --ntasks=1 --gpus=4 --cpus-per-gpu=16 --time=06:00:00

# Component 0: MPI simulation writing snapshots; component 1: GPU model training on them
srun --het-group=0 ./simulate --out "/scratch/$USER/frames" &
srun --het-group=1 python train_surrogate.py --watch "/scratch/$USER/frames" &
wait

sbatch Recipes for AI and HPC

These scripts are complete and use only options confirmed in the Slurm 26.05 and PyTorch documentation. Partition names, accounts, module names and sizes are examples. Replace them with your site's values.

Single GPU

#!/bin/bash
#SBATCH --job-name=finetune-1gpu
#SBATCH --account=mylab            # example
#SBATCH --partition=gpu            # example
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus=1
#SBATCH --cpus-per-gpu=8
#SBATCH --mem-per-gpu=64G
#SBATCH --time=04:00:00
#SBATCH --output=%x-%j.out

module load cuda                   # site-specific
source "$HOME/venvs/torch/bin/activate"

nvidia-smi -L                      # record which GPU the job received
srun python finetune.py --epochs 3 --data "/scratch/$USER/dataset"

Single Node, Multiple GPUs with torchrun

Run one Slurm task per node and let torchrun start one worker per GPU. --standalone sets up a local rendezvous, and SLURM_GPUS_ON_NODE holds the number of GPUs allocated to the batch step.[26][11]

#!/bin/bash
#SBATCH --job-name=ddp-1node
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=4
#SBATCH --cpus-per-task=32
#SBATCH --mem=256G
#SBATCH --time=12:00:00
#SBATCH --output=%x-%j.out

source "$HOME/venvs/torch/bin/activate"
export OMP_NUM_THREADS=8

srun torchrun --standalone --nnodes=1 --nproc-per-node="$SLURM_GPUS_ON_NODE" \
    train.py --config configs/base.yaml

Multiple Nodes with torchrun, srun and c10d Rendezvous

For multi-node jobs, torchrun needs a shared rendezvous ID, a backend (c10d is the recommended one), and an endpoint in host:port form. Start it with the same arguments on every node.[26] Slurm supplies the pieces: the first hostname from scontrol show hostnames, the job ID as the rendezvous ID, and one task per node from srun.

#!/bin/bash
#SBATCH --job-name=ddp-4node
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=96
#SBATCH --mem=0                    # all memory on each node
#SBATCH --exclusive
#SBATCH --time=24:00:00
#SBATCH --output=%x-%j.out

source "$HOME/venvs/torch/bin/activate"

GPUS_PER_NODE=8
head_node=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
export OMP_NUM_THREADS=8
export NCCL_DEBUG=WARN             # use INFO when debugging transport selection

srun --kill-on-bad-exit=1 torchrun \
    --nnodes="$SLURM_JOB_NUM_NODES" \
    --nproc-per-node="$GPUS_PER_NODE" \
    --rdzv-id="$SLURM_JOB_ID" \
    --rdzv-backend=c10d \
    --rdzv-endpoint="${head_node}:29500" \
    train.py --config configs/large.yaml

--mem=0 grants all of each node's memory, and --kill-on-bad-exit=1 ends the step when any task fails, so a dead node does not leave the other workers hanging until walltime.[11][13] NCCL_DEBUG accepts levels including VERSION, WARN and INFO.[27] Add --segment or --switches if your site runs a topology plugin. Fabric tuning (NCCL environment, GPUDirect RDMA) is in Interconnects.

MPI with One Rank per GPU

For CUDA-aware MPI codes, give each rank one GPU with --gpus-per-task=1. That requires an explicit task count and implies per-task binding.[11] Launch with srun --mpi=pmix if Slurm was built with PMIx (srun --mpi=list shows what is available).[13]

#!/bin/bash
#SBATCH --job-name=cfd-gpu
#SBATCH --partition=gpu
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-task=1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-gpu=120G
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out

module load openmpi cuda           # site-specific, CUDA-aware MPI build
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"

srun --mpi=pmix --gpu-bind=verbose,per_task:1 ./solver input.nml

Job Array Hyperparameter Sweep

#!/bin/bash
#SBATCH --job-name=lr-sweep
#SBATCH --partition=gpu
#SBATCH --array=0-23%6             # 24 runs, at most 6 at a time
#SBATCH --ntasks=1
#SBATCH --gpus=1
#SBATCH --cpus-per-gpu=8
#SBATCH --mem-per-gpu=48G
#SBATCH --time=02:00:00
#SBATCH --output=logs/%x_%A_%a.out # create logs/ before submitting

source "$HOME/venvs/torch/bin/activate"

LRS=(1e-4 3e-4 1e-3)
SEEDS=(0 1 2 3 4 5 6 7)
lr=${LRS[$(( SLURM_ARRAY_TASK_ID % 3 ))]}
seed=${SEEDS[$(( SLURM_ARRAY_TASK_ID / 3 ))]}

srun python train.py --lr "$lr" --seed "$seed" --out "runs/lr${lr}_seed${seed}"

Interactive GPU Session

With LaunchParameters=use_interactive_step, salloc gives you a shell on the allocated compute node, using the default --interactive --preserve-env --pty $SHELL step.[12] Without it, the shell opens wherever you ran salloc, and you need srun --pty to reach the node.[13]

# Allocate one GPU for an hour on an interactive partition (names are examples)
salloc --partition=gpu-interactive --gpus=1 --cpus-per-gpu=8 --mem-per-gpu=32G --time=01:00:00

# If your site does not enable use_interactive_step, step onto the node explicitly:
srun --pty bash -l

# One-liner alternative without salloc:
srun --partition=gpu-interactive --gpus=1 --cpus-per-gpu=8 --time=01:00:00 --pty bash -l

Field note: Interactive GPU sessions are the biggest source of idle accelerators on most campus clusters. Put them on a dedicated partition or QOS with a short MaxWall and a per-user GPU cap, and steer notebook users to Open OnDemand, where sessions end when the time runs out instead of when someone remembers to type exit.

Kubernetes for AI Workloads

Device Plugins and Dynamic Resource Allocation

Kubernetes exposes GPUs through the device plugin framework, which has been stable since v1.26. A vendor plugin advertises an extended resource such as nvidia.com/gpu, and pods request it under resources.limits. Extended resources are integers, cannot be overcommitted, and a device cannot be shared between containers.[28] Dynamic Resource Allocation (DRA) is the newer model: drivers publish ResourceSlices, administrators define DeviceClasses, and workloads claim devices through ResourceClaims with CEL filtering on device attributes. DRA has been stable since Kubernetes v1.35 (first available in v1.30). One documented limitation matters for shared clusters: the scheduler does not support preemption for DRA resources.[29] As of September 2026 the supported Kubernetes minors are 1.37, 1.36 and 1.35, and 1.37.0 was released on 2026-08-26.[30]

apiVersion: v1
kind: Pod
metadata:
  name: cuda-smoke-test
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: nvidia/cuda:12.4.1-base-ubuntu22.04   # example image tag (Docker Hub)
    command: ["nvidia-smi", "-L"]
    resources:
      limits:
        nvidia.com/gpu: 1

NVIDIA GPU Operator

Hand-installing drivers, container runtime hooks and the device plugin on every node does not scale. The GPU Operator automates the NVIDIA stack on Kubernetes: drivers, the device plugin, the NVIDIA Container Toolkit, GPU Feature Discovery node labels, DCGM-based monitoring and more.[31] It also deploys a MIG Manager, with a single strategy (MIG on all GPUs of a node) or a mixed strategy.[32] The latest release at the time of writing is v26.7.1, published 2026-09-23.[33] Driver and container details are in Systems Software.

Kueue

Plain Kubernetes has no queue: a job that does not fit just leaves pods pending, and there is no fairshare between teams. Kueue fills that gap. It is a Kubernetes-native system that decides when a job waits, when it is admitted (its pods may be created), and when it is preempted. It deliberately leaves pod placement to kube-scheduler and scaling to the cluster autoscaler.[7] Its core objects map neatly onto HPC concepts:

• ResourceFlavor: a class of nodes (for example a GPU model), selected by node labels and taints.
• ClusterQueue: a quota per flavor (nominalQuota), with borrowingLimit and lendingLimit inside a cohort of queues that share idle capacity.[34]
• LocalQueue: a namespaced entry point that points at a ClusterQueue. Users submit jobs here.[35]

Kueue also offers fair sharing, preemption policies, all-or-nothing (gang) admission, MultiKueue for multi-cluster dispatch, and Topology-Aware Scheduling (beta since v0.14), where annotations such as kueue.x-k8s.io/podset-required-topology keep a job's pods inside a rack or block.[7][36] The current release line is v0.19.[37]

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: h100
spec:
  nodeLabels:
    gpu-type: h100            # a label you apply to the H100 nodes (example)
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: lab-a-cq
spec:
  namespaceSelector: {}
  cohortName: "research"
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: "h100"
      resources:
      - name: "cpu"
        nominalQuota: 256
      - name: "memory"
        nominalQuota: 2048Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 16
        borrowingLimit: 16
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: lab-a
  name: lab-a-queue
spec:
  clusterQueue: lab-a-cq
---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: finetune-
  namespace: lab-a
  labels:
    kueue.x-k8s.io/queue-name: lab-a-queue
spec:
  parallelism: 1
  completions: 1
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: train
        image: registry.example.edu/lab-a/train:2026.09   # example image
        command: ["python", "finetune.py"]
        resources:
          requests:
            cpu: "32"
            memory: 256Gi
          limits:
            nvidia.com/gpu: 4

The job only needs the kueue.x-k8s.io/queue-name label. Kueue suspends it through a webhook and releases it when quota allows.[38]

Volcano and KubeRay

Volcano is a CNCF incubating batch scheduler that replaces or supplements the default scheduler. It provides gang scheduling, heterogeneous device scheduling and topology-aware scheduling, and it integrates with frameworks including PyTorch, Spark, Ray and MindSpore.[8] Choose Kueue if you want to keep kube-scheduler and add queuing on top. Choose Volcano if you want a batch-oriented scheduler in its place.

KubeRay is the operator for Ray on Kubernetes. It provides RayCluster (long-lived clusters with autoscaling), RayJob (creates a cluster, runs a job, optionally tears the cluster down), RayService (Ray Serve with zero-downtime upgrades) and RayCronJob.[39] RayJob works well for batch inference and data processing. RayService belongs with the serving patterns in Inference at Scale.

Slinky: Slurm and Kubernetes Together

SchedMD's Slinky projects connect the two schedulers in two different ways:[6]

• slurm-operator runs Slurm clusters on Kubernetes. Compute nodes are pods grouped into NodeSets, login nodes into LoginSets, and the controller supports HA and zero-downtime reconfiguration. The operator drains Slurm nodes before scale-in or upgrade. Current 1.x releases support Kubernetes 1.35 through 1.37 and require Slurm 25.11 or later.[1]
• slurm-bridge uses Slurm as a Kubernetes scheduler. Kubernetes Pods, Jobs, JobSets and LeaderWorkerSets become placeholder Slurm jobs, so one controller arbitrates both kinds of work. On hybrid nodes, the hardware is shared over time: a node does not run native Slurm jobs and bridge-managed Kubernetes jobs simultaneously.[40]

Slurm vs Kubernetes vs Both

For most university centers the practical answer in 2026 is Slurm for batch, a modest Kubernetes cluster for services and inference endpoints, and a clear rule about which GPUs belong to which. Sites with strong platform teams are starting to evaluate Slinky to merge the two. It is a real option now, but it adds Kubernetes expertise to your staffing needs (see Research Computing Operations). Cloud bursting options are in Cloud and Hybrid.

GPU Sharing: MIG, MPS, Time-Slicing and Shards

Many research jobs (notebooks, small inference, teaching labs, CI) cannot saturate a modern data center GPU. There are four ways to share one, with very different isolation:

MIG splits supported NVIDIA GPUs (Ampere and later) into up to seven GPU instances, each with its own path through the memory system: crossbar ports, L2 banks, memory controllers and DRAM address buses. Throughput and latency stay predictable regardless of the neighbors.[41] Slurm has supported MIG since 21.08. MIG types are named by a substring of the profile string that slurmd reports, e.g. Gres=gpu:a100:1,gpu:a100_3g.20gb:2.[9] In Kubernetes, the GPU Operator exposes MIG profiles as resources like nvidia.com/mig-1g.10gb.[32]

MPS lets cooperating CUDA processes share a GPU with reduced context switching, SM and memory partitioning, and priorities. NVIDIA's 2026 documentation adds MPS v3, an opt-in control interface with scriptable commands and named servers.[42] In Slurm, gres/mps counts are split across a node's GPUs (Gres=gpu:2,mps:200 gives 100 per GPU), a request must be satisfied from a single GPU per node, and a prolog starts and stops the MPS server. Slurm also warns that only one user can have an active MPS server on a system at a time, so different users are time-sliced rather than truly concurrent.[9]

Time-slicing in Kubernetes advertises each GPU as several replicas. A setting of 10 replicas on an 8-GPU node advertises 80 nvidia.com/gpu. Nothing isolates the replicas: they share memory and a fault domain, so if one crashes they all do. Time-slicing and MPS are mutually exclusive, and the same method applies to all GPUs on a node.[43] DCGM-Exporter cannot attribute metrics to containers when time-slicing is on.[44]

# NVIDIA device plugin sharing config (example): 4 replicas per GPU
version: v1
sharing:
  timeSlicing:
    renameByDefault: true      # advertise as nvidia.com/gpu.shared
    resources:
    - name: nvidia.com/gpu
      replicas: 4

Slurm shards (gres/shard) are the Slurm counterpart to time-slicing: several jobs, even from different users, share a GPU up to a configured count, but processes are not fenced. SchedMD therefore recommends shards for homogeneous workloads. A GPU allocated as a whole gpu is not available as shards, and vice versa.[9] A node line like Gres=gpu:l40s:4,shard:64 with no shard lines in gres.conf spreads 16 shards per GPU. Users then request --gres=shard:4.[9]

Field note: Match the sharing method to the trust boundary. MIG is the only option here that isolates memory and faults in hardware, so use it wherever strangers share a card (courses, public notebook services). Shards and time-slicing are fine for one lab's own inference fleet, where a crash only hurts the people who caused it.

Portals and Workflow Managers

Open OnDemand for Jupyter, VS Code and RStudio

Open OnDemand (OOD) is the web portal most research centers now put in front of Slurm. Its interactive apps are batch jobs with a browser front end. Each app has five parts: a manifest, a user form (form.yml.erb), a render template, a job submission file (submit.yml.erb) and a connection view.[45] The community maintains apps for Jupyter, RStudio Server and Visual Studio Code (code-server) from OSC, Utah, Stanford and others that you can adapt instead of writing from scratch.[46]

The job submission file is where GPU policy goes. OOD's portable fields cover common parameters, and the native array passes raw sbatch arguments when you need Slurm-specific options.[47]

# submit.yml.erb for a GPU Jupyter app (example; num_gpus and hours come from form.yml.erb)
---
batch_connect:
  template: "basic"
script:
  queue_name: "gpu-interactive"
  wall_time: <%= (hours.blank? ? 1 : hours.to_i) * 3600 %>
  native:
    - "--gpus=<%= num_gpus %>"
    - "--cpus-per-gpu=8"
    - "--mem-per-gpu=32G"

Offer a small menu rather than free-form fields: "1 MIG slice", "1 full GPU", "4 GPUs" and similar choices, each mapped to a partition and QOS with sane limits.

Workflow Managers: Nextflow, Snakemake, Parsl and Ray

Workflow managers turn a DAG of tasks into many scheduler jobs. On a shared cluster they need throttling, or they will swamp the controller.

• Nextflow submits each task with sbatch when process.executor = 'slurm'. The Slurm executor documents the clusterOptions, cpus, memory, queue and time directives.[48] Pass GPU requests through clusterOptions. The separate accelerator directive is platform-specific, and Nextflow also offers an array directive that batches tasks into Slurm job arrays.[49]
• Snakemake uses the snakemake-executor-plugin-slurm plugin (snakemake --executor slurm --jobs N). Rules request GPUs with gpu=2 and gpu_model="a100" resources, and --slurm-array-jobs submits eligible rules as arrays.[50]
• Parsl parallelizes Python functions as futures.[51] Its SlurmProvider requests blocks of nodes (with gpus_per_node, and exclusive=True by default). The HighThroughputExecutor option available_accelerators pins each worker to one accelerator.[52]
• Ray expects a head node and workers. Ray 2.49 added ray symmetric-run, which starts Ray on every node of a Slurm allocation and runs the entrypoint only on the head. The Ray docs note it does not currently work in multi-tenant setups, where you must assign distinct ports per job.[53]

// nextflow.config (example): CPU tasks to 'cpu', labelled GPU tasks to 'gpu'
process {
    executor = 'slurm'
    queue    = 'cpu'
    withLabel: 'gpu' {
        queue          = 'gpu'
        clusterOptions = '--gpus=1 --cpus-per-gpu=8'
    }
}

# Parsl config (example): up to 4 single-node GPU blocks, one worker per GPU
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.providers import SlurmProvider

config = Config(executors=[
    HighThroughputExecutor(
        label="gpu_htex",
        available_accelerators=4,
        provider=SlurmProvider(
            partition="gpu",
            account="mylab",
            nodes_per_block=1,
            gpus_per_node=4,
            init_blocks=0,
            max_blocks=4,
            walltime="04:00:00",
            worker_init="source ~/venvs/parsl/bin/activate",
        ),
    )
])

# Ray on Slurm with symmetric-run (from the Ray docs pattern; Ray 2.49+)
#SBATCH --nodes=4
#SBATCH --exclusive
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --gpus-per-task=4

GPUS_PER_NODE=4                    # match --gpus-per-task above
head_node=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
ip_head="${head_node}:6379"

srun --nodes="$SLURM_JOB_NUM_NODES" --ntasks="$SLURM_JOB_NUM_NODES" \
    ray symmetric-run --address "$ip_head" --min-nodes "$SLURM_JOB_NUM_NODES" \
    --num-cpus="$SLURM_CPUS_PER_TASK" --num-gpus="$GPUS_PER_NODE" \
    -- python -u batch_infer.py

Queue Policy Design for Mixed AI and Simulation Workloads

Most converged clusters serve three populations at once: simulation groups with long, wide, predictable MPI jobs; AI groups with everything from notebooks to multi-node training; and a long tail of students running one-GPU jobs. A policy that works for all three usually has these parts.

Partitions by Hardware, QOS by Behavior

Make partitions describe hardware pools (CPU, training GPUs, shared or MIG GPUs, large memory) and use QOS to describe behavior (debug, normal, long, hero, scavenger). Policy then lives in the accounting database and changes without daemon restarts. A typical GPU-side ladder:

Principles That Hold Up

• Charge for what is stranded. Weight GPUs in TRESBillingWeights so fairshare reflects scarcity, and set per-GPU defaults (DefCpuPerGPU, DefMemPerGPU) plus QOS limits such as MaxTRESPerNode so one-GPU jobs cannot eat all the cores on an 8-GPU node.[10][12][20]
• Keep simulation off the GPUs unless it uses them. Do not add GPU nodes to the general CPU partition. Instead, let CPU-only work reach idle GPU nodes only through a preemptible QOS.
• Reward accurate time limits. Backfill depends on them. --time-min lets Slurm shorten a job's limit to start it earlier,[11] and a short debug QOS lets people test before asking for a day.
• Use topology for big jobs, not small ones. Route large training to a partition with topology/block and --segment, and keep single-node jobs on a pool where fragmentation does not matter.
• Separate sharing pools. Put MIG and shard-enabled nodes in their own partition so whole-GPU jobs never land behind fractional ones.
• Publish the policy and the data. Share wait-time and utilization reports by partition. Slurm 26.05's OpenMetrics GPU allocation statistics make this easier to automate.[3] Monitoring is covered in Observability.

Field note: Review queue policy every semester with a faculty advisory group, and change one thing at a time. Fairshare half-life, GPU billing weight and per-user caps interact, and if you move all three at once you will not know which change fixed (or caused) the complaint.

Key Takeaways

• Slurm 26.05 is current. It adds ring and torus topologies, async steps, memory resizing and GPU metrics, and NVIDIA's ownership of SchedMD comes with a stated commitment to keep Slurm open source and vendor-neutral.
• Use AutoDetect in gres.conf, cgroup v2 with ConstrainDevices=yes, and track both generic and typed GPU TRES. Without device constraints there is no GPU isolation.
• Know the request options: --gpus-per-node for torchrun-style jobs, --gpus-per-task for MPI rank-per-GPU, --cpus-per-gpu and --mem-per-gpu to size the rest of the job.
• Topology plugins (tree with --switches, block with --segment) matter for multi-node training, but only if the topology file matches the real fabric.
• Weight GPUs heavily in billing, build QOS tiers including a preemptible scavenger tier, and require checkpoint-on-SIGTERM for anything preemptible.
• Kubernetes GPU scheduling means the GPU Operator plus Kueue (or Volcano) for queuing. DRA is stable since v1.35, and Slinky lets Slurm and Kubernetes share one GPU pool.
• Choose GPU sharing by trust boundary: MIG for hardware isolation, MPS or shards and time-slicing only among cooperating users.
• Open OnDemand and workflow managers are part of the scheduling design. Give them curated resource menus and array or throttle settings so they help the queue instead of flooding it.

Sources

1. slurm-operator README (SlinkyProject, GitHub)
2. Slurm version 26.05 is now available, slurm-users mailing list (SchedMD, May 2026)
3. Release Notes for Slurm 26.05 (SchedMD, 2026)
4. NVIDIA Acquires Open-Source Workload Management Provider SchedMD (NVIDIA, December 2025)
5. gres.conf man page (SchedMD, Slurm 26.05 documentation)
6. Slinky (SchedMD, Slurm documentation)
7. Kueue Overview (Kubernetes SIGs)
8. Volcano Introduction (Volcano project, CNCF)
9. Generic Resource (GRES) Scheduling (SchedMD, Slurm 26.05 documentation)
10. Trackable RESources (TRES) (SchedMD, Slurm 26.05 documentation)
11. sbatch man page (SchedMD, Slurm 26.05 documentation)
12. slurm.conf man page (SchedMD, Slurm 26.05 documentation)
13. srun man page (SchedMD, Slurm 26.05 documentation)
14. cgroup.conf man page (SchedMD, Slurm 26.05 documentation)
15. Topology Guide (SchedMD, Slurm 26.05 documentation)
16. topology.conf man page (SchedMD, Slurm 26.05 documentation)
17. Multifactor Priority Plugin (SchedMD, Slurm 26.05 documentation)
18. Fair Tree Fairshare Algorithm (SchedMD, Slurm 26.05 documentation)
19. Quality of Service (QOS) (SchedMD, Slurm 26.05 documentation)
20. sacctmgr man page (SchedMD, Slurm 26.05 documentation)
21. Preemption (SchedMD, Slurm 26.05 documentation)
22. Advanced Resource Reservation Guide (SchedMD, Slurm 26.05 documentation)
23. scontrol man page (SchedMD, Slurm 26.05 documentation)
24. Job Array Support (SchedMD, Slurm 26.05 documentation)
25. Heterogeneous Job Support (SchedMD, Slurm 26.05 documentation)
26. torchrun (Elastic Launch) (PyTorch documentation)
27. NCCL Environment Variables (NVIDIA NCCL User Guide)
28. Device Plugins (Kubernetes documentation)
29. Dynamic Resource Allocation (Kubernetes documentation)
30. Kubernetes Releases (Kubernetes project, 2026)
31. About the NVIDIA GPU Operator (NVIDIA documentation)
32. GPU Operator with MIG (NVIDIA documentation)
33. NVIDIA GPU Operator releases (NVIDIA, GitHub)
34. Kueue Concepts: Cluster Queue (Kubernetes SIGs)
35. Kueue Concepts: Local Queue (Kubernetes SIGs)
36. Kueue: Topology-Aware Scheduling (Kubernetes SIGs)
37. Kueue releases (Kubernetes SIGs, GitHub)
38. Kueue: Run a Kubernetes Job (Kubernetes SIGs)
39. Getting Started with KubeRay (Ray documentation)
40. slurm-bridge README (SlinkyProject, GitHub)
41. Multi-Instance GPU User Guide: Introduction (NVIDIA)
42. Multi-Process Service (NVIDIA documentation, 2026)
43. NVIDIA device plugin for Kubernetes, README (NVIDIA, GitHub)
44. Time-Slicing GPUs in Kubernetes (NVIDIA GPU Operator documentation)
45. Interactive Apps (Open OnDemand documentation)
46. Install Other Interactive Apps (Open OnDemand documentation)
47. Interactive Apps: Job Submission, submit.yml.erb (Open OnDemand documentation)
48. Nextflow SLURM executor (Seqera, Nextflow documentation)
49. Nextflow process reference: accelerator and array directives (Seqera)
50. Snakemake executor plugin: slurm (Snakemake plugin catalog)
51. Parsl: Parallel Scripting Library, README (Parsl project, GitHub)
52. Parsl SlurmProvider and HighThroughputExecutor source documentation (Parsl project, GitHub)
53. Deploying on Slurm (Ray documentation)