Cloud and Hybrid AI/HPC

The 2019 edition of this chapter showed how to stand up an AWS cluster in ten minutes with CfnCluster. The ten-minute part is still true; the tools are better and the instance types are unrecognizable. What changed more is the economics. In 2019 the cloud question was mostly about elastic CPU capacity. In 2026 it is about access to scarce GPUs and high-bandwidth fabrics, and about whether a research group can get 64 GPUs next Tuesday for a week without buying a rack. This chapter covers the cluster tooling on the three large providers, the GPU and TPU offerings with specs taken from the providers' own documentation, the capacity models that decide whether you actually get the hardware, cost and security guardrails for research, hybrid bursting from a campus cluster, and how to think through cloud versus on-prem without inventing prices. The author runs UC Riverside's Ursa Major research computing service on Google Cloud with Cluster Toolkit, so some of the field notes come from operating exactly this kind of environment.

Cloud HPC in 2026

Every large provider now sells three different things that are easy to confuse:

• Raw instances. GPU virtual machines or bare-metal nodes you provision yourself. Maximum control, maximum operational work.
• Cluster toolkits and managed schedulers. Tools that assemble instances, a scheduler (almost always Slurm), shared storage and networking into something that looks like a campus cluster. This is where most research computing teams should start.
• Managed AI platforms. Training and inference services, notebooks and model endpoints. Useful, but they hide the cluster and are harder to bring into a standard research workflow or a Slurm accounting system.

Cloud is also now a genuine capability tier, not just an overflow tank. Microsoft's Eagle system (ND H100 v5, NVIDIA InfiniBand NDR) was No. 5 on the November 2025 TOP500 list and No. 7 in June 2026 at 561.20 PFlop/s HPL.[1] Very large GPU systems exist in the cloud; the question for a university is how to get a useful slice of one at a price and with a process that fits research funding.

Cluster Tooling on the Big Three

AWS ParallelCluster and AWS PCS

AWS ParallelCluster is the descendant of CfnCluster. You describe the cluster in YAML, run pcluster create-cluster, and it builds a head node and auto-scaling Slurm queues in your account. Version 3.16.1 (September 2026) ships Slurm 25.11.8.[2] Queues support a CapacityType of ONDEMAND, SPOT or CAPACITY_BLOCK; with Capacity Blocks, MinCount must equal MaxCount because the reserved instances are managed as static nodes.[3] A sketch of a GPU queue alongside a spot CPU queue (IDs and counts are placeholders):

Region: us-east-1                      # example
Image:
  Os: alinux2023
HeadNode:
  InstanceType: c5.xlarge
  Networking:
    SubnetId: subnet-0123456789abcdef0   # placeholder
  Ssh:
    KeyName: mykey                       # placeholder
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: cpu-spot
      CapacityType: SPOT
      ComputeResources:
        - Name: c5-2xl
          InstanceType: c5.2xlarge
          MinCount: 0
          MaxCount: 20
      Networking:
        SubnetIds:
          - subnet-0123456789abcdef0
    - Name: gpu
      CapacityType: ONDEMAND
      ComputeResources:
        - Name: p5
          InstanceType: p5.48xlarge
          MinCount: 0
          MaxCount: 2
          Efa:
            Enabled: true
      Networking:
        SubnetIds:
          - subnet-0123456789abcdef0
        PlacementGroup:
          Enabled: true

AWS Parallel Computing Service (PCS) is the managed alternative: AWS runs the Slurm controller, and you define compute node groups and queues through the service API; PCS launches the EC2 instances in your account and bills you for them.[4] PCS publishes a support calendar for Slurm versions; as of September 2026 it supports Slurm 25.05, 25.11 and 26.05, with each version carrying an end-of-life date.[5] Choose ParallelCluster when you want full control and are comfortable owning the head node; choose PCS when you would rather not patch and upgrade a Slurm controller yourself.

Google Cloud Cluster Toolkit

Cluster Toolkit (formerly HPC Toolkit) is open-source. A blueprint is a YAML file that lists modules (network, file systems, Slurm node sets and partitions, login and controller nodes). The gcluster tool compiles a blueprint into a deployment folder of Terraform and Packer configuration, which you then deploy.[6] The basic lifecycle is gcluster create, gcluster deploy and gcluster destroy.[7] An abbreviated Slurm blueprint in the style of the project's hpc-slurm.yaml example:[8]

blueprint_name: research-slurm
vars:
  project_id: my-research-project        # placeholder
  deployment_name: research-slurm
  region: us-central1
  zone: us-central1-a

deployment_groups:
- group: primary
  modules:
  - id: network
    source: modules/network/vpc

  - id: homefs
    source: modules/file-system/filestore
    use: [network]
    settings:
      local_mount: /home

  - id: compute_nodeset
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [network]
    settings:
      node_count_dynamic_max: 20
      machine_type: h3-standard-88
      enable_spot_vm: true               # example: preemptible CPU partition

  - id: compute_partition
    source: community/modules/compute/schedmd-slurm-gcp-v6-partition
    use: [compute_nodeset]
    settings:
      partition_name: compute

  - id: slurm_login
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-login
    use: [network]
    settings:
      machine_type: n2-standard-4

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
    use: [network, compute_partition, homefs, slurm_login]

Node sets expose the capacity knobs directly: enable_spot_vm, reservation_name for a specific reservation, and dws_flex to provision through Dynamic Workload Scheduler flex-start (with a maximum run duration that should not exceed one week).[9] Google also ships GPU-focused blueprints such as ml-slurm.yaml and GKE blueprints for A3 machines.[10]

Field note: Treat blueprints as code. Keep them in Git, parameterize project and region, and review changes like any other pull request. At UCR we can rebuild a cluster from the blueprint when something drifts, which is far cheaper than debugging a hand-modified one. The deployment folder is generated output; do not hand-edit it and then lose the changes on the next gcluster create.

Azure CycleCloud

Azure CycleCloud is an orchestration application that runs on a VM in your subscription and manages clusters. Azure CycleCloud Workspace for Slurm is a Marketplace solution template that deploys the network, a CycleCloud VM, a shared file system for home directories, optional Azure Managed Lustre, and preconfigured Slurm clusters with PMIx v4, Pyxis and enroot for containerized jobs.[11] Microsoft's own caveat is worth repeating: deploying the template needs no prior Azure or Slurm knowledge, but operating and customizing it does.[11]

GPU Instance Families

All three providers sell 8-GPU HGX-style nodes with a dedicated RDMA fabric, and all now sell GB200 NVL72 rack-scale systems carved into smaller VMs. The table uses each provider's documentation for GPU type, count, memory and scale-out network bandwidth. Scale-up (NVLink) details are in Interconnects, and the GPUs themselves in Accelerators.

Notes on the table. Google's figures for A3 Ultra, A4 and A4X Max are maximum total instance bandwidth; the dedicated GPU-to-GPU portion on A3 Ultra and A4 is eight ConnectX-7 NICs at 3,200 Gbps, and on A4X Max it is four ConnectX-8 SuperNICs at 3,200 Gbps.[15] The A3 Mega, High and Edge types use GPUDirect-TCPXO or GPUDirect-TCPX rather than RDMA, which matters for tightly coupled MPI codes.[15] AWS lists P6-B300 as 1.5x the GPU memory and 2x the network bandwidth of P6-B200.[13] Azure's GB200 v6 page writes the Quantum-2 CX7 InfiniBand links as "400 GB/s"; the same page gives 28.8 Tb/s of scale-out networking per 72-GPU rack, which works out to 400 Gb/s per GPU, so the table uses Gb/s.[19]

Two practical points. First, the cloud network name tells you what the fabric really is: AWS EFA is AWS's own RDMA transport (libfabric), Azure ND uses real InfiniBand, and Google uses RoCE-based GPUDirect RDMA on A3 Ultra and newer. MPI and NCCL builds must match the fabric; use the provider's images or documented stacks. Second, rack-scale NVL72 systems are sold as groups of VMs: AWS P6e-GB200 instances are only available in UltraServers, Azure describes a GB200 NVL72 rack as 18 ND GB200 v6 VMs, and Google's A4X scales NVLink up to 72 GPUs in one domain.[13][19][16]

TPU Pods on Google Cloud

Google's TPUs are the main non-NVIDIA option at pod scale in a public cloud. They are programmed through JAX, PyTorch/XLA or similar XLA-based frameworks rather than CUDA, so they suit groups whose code is already in JAX or who are training transformer-style models. Current generations, from Google's documentation:[20]

All figures are Google's stated peak values from the TPU7x and TPU machine documentation.[20][21] On v5p the largest schedulable job is 6,144 chips out of the 8,960-chip pod.[22] TPUs are a poor fit for classic FP64 simulation codes. For AI for science, they matter: GenCast inference, for example, was characterized on a Cloud TPU v5 device (see Scientific Computing).

Capacity Models: Getting the GPUs

For CPU work, cloud capacity is effectively unlimited. For current-generation GPUs, it is not, and the purchase model decides whether your job starts.

Field note: Spot GPUs are the cheapest hours in the cloud and the easiest to waste. A job that checkpoints every few hours and gets preempted every hour makes no progress at all. Before you put users on spot GPU partitions, make sure their codes checkpoint on a signal or at short intervals, that Slurm requeues preempted jobs, and that checkpoints go to durable storage, not local SSD that vanishes with the VM.

A pattern that works well in research: run interactive and development work on small on-demand or spot nodes, and book Capacity Blocks or calendar-mode reservations for the handful of large runs each term. The reservation start date becomes a project deadline, which helps groups get their data staged and their code tested before the expensive hours begin.

Neoclouds and GPU Specialists

A set of GPU-focused providers (often called neoclouds) sells large NVIDIA clusters with InfiniBand and Slurm or Kubernetes on top, usually with fewer surrounding services than the hyperscalers and often with simpler pricing. Examples, described from their own documentation:

• CoreWeave offers SUNK (Slurm on Kubernetes), a Helm-deployed Slurm running inside its Kubernetes service so training and inference can share the same nodes.[30]
• Lambda 1-Click Clusters provide 16 to 512 NVIDIA H100 or B200 GPUs on a 400 Gb/s Quantum-2 InfiniBand rail-optimized fabric, with GPUDirect RDMA up to 3,200 Gb/s per node.[31]
• Nebius offers VMs clustered over InfiniBand, managed Kubernetes and Soperator, its Kubernetes operator for Slurm.[32]

Neoclouds can be a good fit for a single well-defined training or simulation campaign with a grant budget attached. Check the things hyperscalers bundle and neoclouds may not: identity federation with your campus, data egress terms, storage durability, compliance attestations, and contract terms your procurement office will accept. Kubernetes-hosted Slurm is covered in Scheduling and Orchestration.

Cost Management and Guardrails for Research

The fastest way to lose faculty trust in cloud is a surprise bill. Guardrails must be in place before the first GPU node starts.

Credits and subsidized access

• Provider research credits. Google Cloud advertises research credit applications (currently $5,000 per application on its researcher page), and AWS runs the Cloud Credit for Research program.[33][34] Credits expire and are tied to accounts; track them like any other award.
• NSF programs. CloudBank provides no-cost, NSF-sponsored access to commercial clouds through ACCESS and the NAIRR Pilot, along with account management and cost monitoring.[35] The NAIRR Pilot accepts requests from US-based researchers and educators for computing, model and platform resources for AI research, with 12-month awards.[36]

Budgets, quotas and hard stops

• Budgets alert; they do not stop spending by default. Google says plainly that an alerts-only budget does not cap usage, and offers programmatic notifications (Pub/Sub) that can disable billing on a project, plus a spend-cap budget type in preview.[37] AWS Budgets can run actions when a threshold is crossed, including applying an IAM policy or service control policy, or targeting specific EC2 or RDS instances.[38]
• Quotas are your cheapest hard limit. A project with a GPU quota of 8 cannot run 64 GPUs by accident. Keep GPU quotas at what a project actually needs and raise them deliberately.
• Scheduler limits still apply. In a cloud Slurm cluster, the maximum node count per partition (for example MaxCount in ParallelCluster or node_count_dynamic_max in Cluster Toolkit) is a spending cap. Slurm accounting (QOS, association limits) gives per-group control, exactly as on campus.
• One project or account per grant. Separate billing projects per award make chargeback trivial and let you shut down one group without touching others. Label or tag every resource with owner and fund.

Field note: The most common runaway costs we see are not big GPU jobs. They are things left running: a login node sized for a workshop, a forgotten notebook VM with a GPU attached, a Filestore or FSx file system for a project that ended, or snapshots nobody deletes. Schedule a weekly report of running resources per project and send it to the PI, not just to IT.

Egress and storage costs

Compute is billed while it runs; storage is billed forever, and moving data out is billed per byte. For research, three rules cover most cases:

• Ingress is generally free; egress is not. Plan where results end up. Moving a finished multi-terabyte dataset back to campus is a real line item unless a waiver applies.
• Check the research egress waivers. AWS's Global Data Egress Waiver for eligible researchers and institutions waives data transfer out to the internet up to 15 percent of total monthly AWS spend.[39] Google waives internet data transfer fees at list price for qualifying research and education customers, up to 15 percent of total monthly Google Cloud fees.[40] Both have eligibility requirements (typically routing over a national research and education network and an institutional agreement); confirm with your account team.
• Tier storage deliberately. Parallel file systems (FSx for Lustre, Managed Lustre, Filestore and partner products) are for active jobs. Object storage is for everything else. AWS notes that P6 instances pair with FSx for Lustre for high-throughput access and with S3 for bulk storage.[13] Delete or tier scratch when the job ends; lifecycle policies do this automatically.

Data pipelines, staging and parallel file system design are covered in Memory and Storage.

Hybrid Bursting from Campus Slurm

Most universities will run a campus cluster and use cloud for peaks, special hardware and deadlines. There are three common patterns, in rising order of integration:

1. Separate cloud cluster, same user experience. Build a cloud Slurm cluster with the same modules, container images and job script conventions as campus, and give users both. Simple and robust; data movement is manual.
2. Multi-cluster Slurm. Campus and cloud clusters share a Slurm database so users can target either with sbatch --clusters= (-M), and accounting lands in one place.[41] Slurm federation goes further, scheduling jobs peer-to-peer across member clusters by creating sibling jobs.[42]
3. Cloud nodes inside the campus scheduler. Slurm's power saving machinery can manage cloud instances directly: nodes defined with State=CLOUD are created by a ResumeProgram when jobs need them and removed by a SuspendProgram after SuspendTime of idleness; nodes that do not register within ResumeTimeout are marked DOWN and their jobs requeued.[43] This is the same mechanism the cloud toolkits use internally.

A minimal slurm.conf fragment for pattern 3. The programs are scripts you (or a toolkit) provide that call the cloud API; names and sizes are examples.

# Cloud burst partition (example values)
SuspendProgram=/opt/slurm/cloud/suspend.sh
ResumeProgram=/opt/slurm/cloud/resume.sh
ResumeFailProgram=/opt/slurm/cloud/resume_fail.sh
SuspendTime=600          # seconds idle before a cloud node is deleted
SuspendTimeout=300
ResumeTimeout=900        # GPU VMs can take several minutes to boot and register
ResumeRate=20
SuspendRate=20

GresTypes=gpu
NodeName=burst-gpu-[001-016] State=CLOUD CPUs=96 RealMemory=1800000 Gres=gpu:8 Feature=cloud,h100
PartitionName=burst Nodes=burst-gpu-[001-016] MaxTime=2-00:00:00 State=UP

The hard parts of bursting are not the scheduler configuration:

• Data gravity. If a job reads 50 TB from campus storage, bursting it is slow and may be expensive. Burst workloads whose inputs are small, already in the cloud, or can be staged ahead of time.
• Identity and networking. Cloud nodes need the same users, groups and file permissions, and a secure path back to campus (VPN or dedicated interconnect). Plan the IP address space so cloud subnets do not collide with campus ranges.
• Software. Container images (Apptainer or enroot) are the easiest way to guarantee the same stack on both sides. A shared Spack build cache also works.
• Accounting. Decide in advance who pays for burst hours and how they are charged back; Slurm accounting can record them, but a policy has to price them.

Field note: Burst the jobs, not the users. A small number of well-understood, containerized, restartable workloads (parameter sweeps, ensemble members, inference batches) move to cloud easily. Trying to make every campus job cloud-portable is a multi-year project that rarely pays off.

Security and Compliance in the Cloud

Cloud providers run a shared responsibility model: the provider secures the facilities, hardware and virtualization layer, and the customer is responsible for the guest operating system (including patches), application software and the configuration of the provider's security controls.[44] For a research cluster, that means you still own the OS images, Slurm, user accounts, firewall rules and data access policy.

• Identity. Federate cloud consoles and cluster logins with campus identity (SAML or OIDC, with MFA). Avoid long-lived access keys on login nodes; use instance roles or service accounts scoped to the project.
• Network. Keep compute nodes on private subnets. Expose only login nodes or a web portal (such as Open OnDemand), ideally behind campus VPN or an identity-aware proxy.
• Organization-level policy. Use AWS Organizations service control policies, Google Cloud organization policies or Azure Policy to block risky configurations (public buckets, unapproved regions) across every research project.
• Regulated data. Controlled Unclassified Information falls under NIST SP 800-171 (Revision 3 published May 2024).[45] Cloud can host CUI and other regulated data, but only in an environment designed and documented for it, usually a separate enclave with its own accounts, logging and review. Do not let regulated data wander into a general-purpose research project.
• Logging. Turn on audit logs (CloudTrail, Cloud Audit Logs, Azure Activity Log) for every project from day one and send them to a central, write-protected location.

Governance, access and data policy for research computing generally are in Operations.

When Cloud Beats On-Prem, and Vice Versa

This section deliberately contains no prices. List prices change monthly, negotiated research pricing varies by institution, and GPU availability swings the real cost more than the rate card. The reasoning, however, is stable.

The utilization breakeven

An on-prem GPU node has a mostly fixed cost over its life: purchase, installation, power and cooling, data center space, support contracts and staff time, spread over the years you run it. A cloud GPU node has a variable cost: you pay for the hours you use. So the comparison comes down to one number, expected sustained utilization:

on-prem cost per useful GPU-hour =
    (total lifetime cost of the node)
    / (GPUs x hours in service life x utilization)

cloud cost per useful GPU-hour =
    (effective hourly rate after discounts, credits, waivers)
    + (storage and egress attributable to the work)
    + (staff time to operate the cloud environment, per hour)

breakeven utilization U* = the utilization where the two are equal

If you expect to keep the hardware busy well above U*, buying wins. If demand is spiky, uncertain or short-lived, renting wins. Fill in the terms with your institution's real quotes (hardware, facilities rates, negotiated cloud pricing) rather than numbers from a blog post. Two effects push U* around: a shared campus cluster that is always queued is running at high utilization and is hard to beat, while a lab-owned node that sits idle between papers is often more expensive than cloud even at on-demand rates.

Cloud tends to win when

• Demand is bursty or tied to deadlines, conferences or a single grant period.
• You need hardware you cannot get or house: the newest GPU generation, NVL72 racks, TPU pods, or more power and cooling than your data center has (see Facilities).
• The project is short or exploratory, and buying would create stranded hardware.
• Data already lives in the cloud (public datasets hosted by providers, instrument data delivered there).
• Teaching and workshops need many identical environments for a few days.

On-prem tends to win when

• Utilization is high and steady, typically a well-run shared cluster with a queue.
• Data is large and generated or used on campus, so egress and transfer time dominate.
• Jobs are long-running and tightly coupled, and need predictable fabric performance over months.
• Funding comes as capital (equipment grants) rather than operating money, or indirect-cost rules make cloud spending expensive for the PI.
• Regulated data requirements are already met by an existing on-prem enclave.

Field note: Most campuses end up hybrid, and that is the right answer, not a compromise. Own the baseline you can keep busy, rent the peaks and the exotic hardware, and make the two feel the same to users with shared containers, job script conventions and identity. The worst outcome is two disconnected worlds that each require their own training and support.

Key Takeaways

• Start with the cluster tooling, not raw instances: AWS ParallelCluster or PCS, Google Cloud Cluster Toolkit, or Azure CycleCloud Workspace for Slurm all deliver Slurm clusters that feel like campus.
• All three providers offer 8-GPU H100, H200 and B200 nodes with RDMA fabrics and GB200-class rack-scale systems; check the fabric type (EFA, InfiniBand, RoCE or TCPX) before running tightly coupled MPI.
• Getting GPUs is a capacity question. Use Capacity Blocks or DWS calendar reservations for planned runs, flex-start for jobs that can wait to start, and spot only for checkpointed work.
• Budgets alert but do not stop spending by default. Use quotas, scheduler node caps, one project per grant, and automated actions as hard limits.
• Plan data movement: ingress is generally free, egress is not, and both AWS and Google offer research egress waivers up to 15 percent of monthly spend.
• Burst well-understood, containerized, restartable workloads; Slurm's cloud node support (State=CLOUD, ResumeProgram/SuspendProgram) is the underlying mechanism.
• Decide cloud versus on-prem on expected sustained utilization with your own institution's quotes; high, steady load favors owning, spiky or exotic demand favors renting.

Sources

1. Eagle - Microsoft NDv5, NVIDIA H100, InfiniBand NDR (TOP500, 2026)
2. AWS ParallelCluster release notes and document history (AWS, 2026)
3. AWS ParallelCluster Scheduling section: SlurmQueues, CapacityType (AWS, 2026)
4. AWS Parallel Computing Service API Reference: Welcome (AWS, 2026)
5. Slurm versions in AWS PCS (AWS, 2026)
6. Cluster Toolkit overview (Google Cloud, 2026)
7. Cluster Toolkit README (Google Cloud, 2026)
8. Cluster Toolkit example blueprint hpc-slurm.yaml (Google Cloud, 2026)
9. schedmd-slurm-gcp-v6-nodeset module README (Google Cloud Cluster Toolkit, 2026)
10. Cluster Toolkit example blueprints (Google Cloud, 2026)
11. What is Azure CycleCloud Workspace for Slurm? (Microsoft Learn, 2026)
12. Amazon EC2 P5 instances (AWS, 2026)
13. Amazon EC2 P6e UltraServers and P6 instances (AWS, 2026)
14. GPU machine types (Google Cloud Compute Engine, 2026)
15. Network bandwidth and GPU machine types (Google Cloud Compute Engine, 2026)
16. GPU machines in the accelerator-optimized machine family (Google Cloud Compute Engine, 2026)
17. ND-H100-v5 sizes series (Microsoft Learn, 2026)
18. ND-H200-v5 size series (Microsoft Learn, 2025)
19. ND GB200-v6 sizes series (Microsoft Learn, 2026)
20. TPU7x (Ironwood) architecture and specifications (Google Cloud, 2026)
21. TPU machines in the accelerator-optimized machine family (Google Cloud, 2026)
22. TPU v5p (Google Cloud, 2026)
23. Spot Instance interruption notices (AWS, 2026)
24. About Azure Spot Virtual Machines (Microsoft Learn, 2026)
25. Spot VMs (Google Cloud Compute Engine, 2026)
26. Find and purchase Capacity Blocks (AWS, 2026)
27. Capacity Blocks for ML (Amazon EC2 User Guide, 2026)
28. About Flex-start VMs (Google Cloud Compute Engine, 2026)
29. About future reservation requests in calendar mode (Google Cloud Compute Engine, 2026)
30. About SUNK (CoreWeave documentation, 2026)
31. 1-Click Clusters introduction (Lambda documentation, 2026)
32. Slurm and Soperator in Nebius AI Cloud (Nebius documentation, 2026)
33. Google Cloud for researchers: research credits (Google Cloud, 2026)
34. AWS Cloud Credit for Research (AWS, 2026)
35. CloudBank (NSF-funded, 2026)
36. NAIRR Pilot resource requests to advance AI research (NAIRR Pilot, 2026)
37. Create, edit, or delete budgets and budget alerts (Google Cloud Billing, 2026)
38. Configuring budget actions (AWS Cost Management, 2026)
39. Data egress waiver available for eligible researchers and institutions (AWS Public Sector Blog, 2021)
40. Get a data transfer discount for research and education (Google Cloud Billing, 2026)
41. sbatch man page: --clusters (SchedMD, 2026)
42. Slurm Federated Scheduling Guide (SchedMD, 2026)
43. Slurm Power Saving Guide: cloud nodes, ResumeProgram and SuspendProgram (SchedMD, 2026)
44. Shared Responsibility Model (AWS, 2026)
45. NIST SP 800-171 Rev. 3: Protecting Controlled Unclassified Information in Nonfederal Systems (NIST, 2024)