Hardware is the easy part. A research computing center succeeds or fails on the things that never show up on a spec sheet: who can get an account and how fast, what it costs a lab and how that cost is justified, whether a second-year grad student can get a PyTorch job running on a GPU in an afternoon, whether the controlled-data project gets a yes from the security office, and whether the provost believes the investment paid off. AI has raised the stakes on every one of those questions. GPU demand now comes from departments that never used the cluster before, it arrives faster than procurement cycles, and it drags compliance and cost questions along with it. This chapter is the operations playbook: service models, national resources, facilitation, security and compliance, cost recovery, capacity planning, staffing, sustainability and impact. Scheduler mechanics are in Scheduling and Orchestration, and cloud options are in Cloud and Hybrid.
Service Models
Almost every campus service is a blend of four models. The blend is what matters, because each model attracts a different population and creates a different political problem.
Condo (Buy-In) Model
In a condo, faculty buy nodes with grant or startup funds, and the center integrates them into a shared cluster with shared networking, storage, software and staff. Purdue's Community Cluster Program is the long-running example. Purdue affiliates have pooled purchases since Q4 2006, the center handles patching, repairs, data center space, power and cooling, and pooled funds give more vendor leverage than individual labs would get.[1] At UC Riverside, owned nodes get a priority-queueing model: the owning lab has high-priority access, its CPU or GPU quota rises with the capacity it bought, and idle owned capacity can run short jobs from other registered users.[2]
Condos turn grant money into shared infrastructure and give faculty a sense of ownership. The costs are heterogeneity (every procurement brings a new node type), awkward queue structures, and relationship overhead. Georgia Tech's PACE team cites exactly these problems (unsustainable hardware heterogeneity, unstable queue structures, stakeholder management overhead) in explaining why institutions move away from pure condos.[3]
Fee-for-Service and Consumption Models
In a consumption model, the center owns the hardware and researchers pay for what they use, as core-hours, GPU-hours, or credits. Georgia Tech's Phoenix transition is the best-documented recent case. PACE bought all-new hardware during a data center move, ran a rate study covering hardware, effort-weighted salaries, licenses and the data center lease over a five-year period, converted faculty hardware funds into compute credits, and introduced a free tier for all faculty (68 credits per month plus the first TB of project storage). A key decision was waiving facilities and administrative overhead (then 57.8%) on the rate, which leadership accepted because it removed a barrier to adoption.[3] Phoenix also shows that consumption models need ongoing negotiation: after faculty feedback, PACE extended refresh-credit lifetime from 5 to 7.5 years.[3]
Federal cost principles constrain how you set these rates. Under 2 CFR 200.468, costs of specialized service facilities, explicitly including computing facilities, are allowable when charged on actual usage through a rate schedule that does not discriminate between federal and non-federal users, is designed to recover only aggregate costs, and is adjusted at least biennially for over- or under-recovery.[4] Involve your cost accounting office before you publish a GPU-hour rate, not after.
Subscriptions, Allocations and Free Tiers
Many centers use a flat subscription or a centrally funded allocation with quotas. UC Riverside's HPCC charges an annual lab registration fee of $1,000 that covers everyone in the lab. CPU use carries no extra charge but is capped at 384 cores per user and 768 per lab, and storage is sold in 10 TB units ($1,000 per year rented, or $260 per year in support fees for lab-owned drives).[2] Allocation models (proposals reviewed by a faculty committee) are common for large central systems and mirror how national resources work. A free or nearly free entry tier matters more than any other design choice for adoption: it is how new faculty and students in non-traditional disciplines discover they need the cluster at all.
Cloud Credits and Brokered Cloud
Cloud is now a standard part of the service catalog rather than an exception. CloudBank, an NSF-supported service, provides access to commercial clouds for researchers and educators through ACCESS and the NAIRR Pilot, with fast account provisioning, cross-cloud spend monitoring, automated balance alerts and training material.[5] Campus-negotiated agreements and vendor research credit programs fill the rest. The operational lesson is to broker cloud through the center, with guardrails, billing visibility and a clear exit path for data, rather than letting every lab open an account on a personal card. Architecture patterns are in Cloud and Hybrid.
| Model | Who pays | Strength | Watch out for |
|---|---|---|---|
| Condo | Faculty buy nodes | Leverages grants and startup funds | Heterogeneity, "my nodes" politics |
| Consumption | Per core-hour or GPU-hour | Fair, scales with use, simple hardware | Rate studies, F&A treatment, budgeting uncertainty for PIs |
| Subscription | Flat lab fee | Predictable, low friction | Heavy users subsidized by light users |
| Allocation | Institution (central) | Merit-based, no billing | Committee load, demand exceeding supply |
| Cloud credits | Grants, vendors, NSF programs | Elastic, newest hardware | Egress, runaway spend, data governance |
National Resources
No campus can meet all of its researchers' needs locally, and in 2026 most campuses cannot keep up with GPU demand alone. A good center treats national resources as part of its own capacity and helps researchers get there.
NSF ACCESS
ACCESS (Advanced Cyberinfrastructure Coordination Ecosystem: Services and Support) is the NSF program that succeeded the 11-year XSEDE project. NSF awarded $52 million over five years to five lead institutions to run it, and it gives researchers and educators, with or without grants, access to national systems at no cost to them.[6] The program reports that in a given month more than 2.5 million jobs are run on average by more than 4,000 researchers on more than 2,000 projects.[7]
ACCESS has four project types: Explore (evaluation, small classes, code porting), Discover (modest grant-funded research, Campus Champions, larger classes), Accelerate (experienced mid-scale users) and Maximize (large-scale work, reviewed in semi-annual rounds).[8] The first three are awarded in ACCESS Credits, which researchers exchange for specific resources at fixed rates. A credit is set at about one core-hour or 1 GB of storage, GPU-hours cost more credits, and credits arrive in two halves: one at award and one after a progress report.[9] Graduate students can be PIs on requests but need a signed advisor letter.[10]
Field note: Get every new faculty member an Explore project in their first semester, even if the campus cluster meets their needs today. It takes a one-page description, it teaches them the national ecosystem, and it gives you an overflow valve when your GPU queue is full the week before a conference deadline.
NAIRR: From Pilot to Operations Center
The National Artificial Intelligence Research Resource (NAIRR) is an NSF-led effort to give the research and education community access to computing, data, models, software, training and expertise for AI. It was established as a pilot in 2024, working with other federal agencies and non-governmental partners.[11] NIH and DOE lead the NAIRR Secure element, which focuses on privacy- and security-preserving infrastructure for controlled-access data.[11][12]
The pilot is now moving to a sustained program. On September 1, 2026, NSF announced the NSF NAIRR Operations Center (NAIRR-OC), led by the San Diego Supercomputer Center with the Texas Advanced Computing Center. It will coordinate resource providers, run the national NAIRR portal, provide user support and training, and lead the transition of pilot functions into long-term infrastructure. The same announcement says the pilot has supported over 800 research projects with more than 25 private-sector partners.[13] SDSC describes the award as a $35 million cooperative agreement over five years.[14] NSF has also announced State and Regional AI Infrastructure Hubs, up to 10 of which will pool regional AI computing and are encouraged to integrate with NAIRR.[12] For campus centers, the practical message is to watch the NAIRR portal for allocation calls, fold NAIRR into your consultation scripts next to ACCESS, and look at whether a regional hub proposal fits your institution.
DOE INCITE and ALCC
For the largest problems, DOE's leadership computing programs are the path. INCITE is open to researchers from academia, government laboratories and industry, and is the major means by which the scientific community gains access to the nation's fastest supercomputers. New INCITE proposals are accepted between mid-April and June 15 (2026 dates for the 2027 call), and small Director's Discretionary awards, typically less than 75,000 node-hours, are available year-round for porting and benchmarking before a proposal.[15][16] The ASCR Leadership Computing Challenge (ALCC) targets DOE-mission, high-risk high-payoff campaigns. Each year it makes 10% to 30% of resources at NERSC, ALCF and OLCF available, on a July 1 to June 30 allocation year with renewals of up to two years.[17]
NSF's Leadership-Class Computing Facility system, Horizon at TACC, is coming online. TACC's user guide noted in July 2026 that Horizon was still limited to internal users, so treat it as upcoming capacity rather than something to plan current projects around.[18]
Community and Consortia
Research computing is a small profession, and much of what works was learned from peers. Three communities repay the time:
- Campus Champions: a community of research computing and data professionals who connect their campuses to ACCESS and other cyberinfrastructure and help users get start-up allocations. The program lists nearly 800 Champions at more than 360 institutions, runs affinity groups and an AI mentorship program, and is funded in part by NSF as part of the NAIRR Pilot.[19]
- CaRCC (Campus Research Computing Consortium): runs the People Network, with tracks organized around role-based "facings" (Researcher-, Data-, Systems-, and Strategy and Policy-Facing) that each have email lists, Slack channels and monthly calls.[20] CaRCC also maintains the RCD Capabilities Model, developed with Internet2 and EDUCAUSE with NSF support, which institutions use to assess and benchmark their research computing and data support.[21]
- US-RSE: the US Research Software Engineer Association, for people who write research software. Relevant as centers add RSE groups alongside facilitation and systems staff.[22]
Field note: Run the CaRCC Capabilities Model assessment before you write a strategic plan or ask for a budget increase. It gives you a structured inventory of gaps, and the peer benchmarking is the most persuasive slide you can show a dean who asks "what are other schools doing?"
Facilitation and Training
Facilitation is the difference between a cluster and a service. CaRCC's professionalization work defines RCD facilitation as advancing scholarly discovery by empowering researchers to better use research computing and data resources themselves.[23] In practice that means consultations, office hours, onboarding, documentation, workshops and embedded collaboration on hard projects.
The Carpentries and Workshop Programs
The Carpentries is a nonprofit that teaches foundational coding and data science skills to researchers, with Software, Data and Library Carpentry lesson programs and a large volunteer instructor community.[24] HPC Carpentry entered Lesson Program Incubation in July 2024, a step toward joining the other lesson programs.[25] Its Using High-Performance Computing Systems (hpc-intro) lesson is in beta, meaning it is ready for instructors outside the original authors, and is designed to fit in about two days of teaching.[26] Adapting hpc-intro to your cluster (your module names, your partitions, your storage paths) is faster than writing an intro course from scratch, and certified Carpentries instructors on staff raise the quality of all your teaching.
A workshop ladder that works on many campuses:
- Unix shell and SSH basics (Software Carpentry).
- Intro to the cluster: modules, storage, Slurm, job arrays (hpc-intro adapted locally).
- Python or R environments and containers (see Systems Software).
- GPU computing: requesting GPUs, checking utilization, data loading, mixed precision.
- Distributed training and scaling (see Training at Scale).
- Serving and fine-tuning models on shared infrastructure (see Inference at Scale).
Documentation
Documentation is facilitation that scales. Treat it as a product. Keep it in version control, give every page an owner and a review date, and test every code sample against the current cluster after each upgrade (Slurm and CUDA upgrades break copied scripts quietly). Organize it by task ("run a multi-GPU PyTorch job", "move 50 TB from a sequencer") rather than by system component, and put a working example at the top of every page. Your ticket system shows you which pages to write next: if the same question appears three times in a month, it needs a page.
Onboarding Researchers to GPUs
AI has brought a new user population: people who know PyTorch well but have never seen a batch scheduler, and domain scientists who want to fine-tune a model they found online. Most of them arrive with a laptop or cloud notebook workflow. A structured on-ramp saves both sides a lot of time.
A GPU Onboarding Path
- Intake conversation. What model, what data size, what framework, what deadline, and is any data regulated? Ten minutes here routes the user to the right partition, a MIG slice, a national resource, or a secure enclave.
- Interactive start. Open OnDemand Jupyter on a small GPU or MIG instance, with a curated environment that already has CUDA-matched PyTorch.
-
First batch job. A template single-GPU sbatch
script (see Scheduling) with
nvidia-smilogging and checkpointing built in. - Efficiency check. Review the first real job's GPU utilization with the user. Low utilization is almost always a data-loading or CPU-count problem, not a need for more GPUs (see Observability).
- Scale up deliberately. Multi-GPU and multi-node only after single-GPU efficiency is good. Point them to ACCESS or NAIRR if their demand exceeds what local policy allows.
Field note: Gate access to the big GPU partition behind a short efficiency conversation, not a quiz. Researchers accept "show us one job at decent utilization on a single GPU and we will open the 8-GPU queue" because it is obviously fair, and it catches most wasted-GPU problems before they happen.
Security and Compliance
Regulated data used to be a niche for a few medical and defense labs. AI has changed that: clinical notes, genomic data, imaging, student records and export-controlled engineering data are all now candidate training data. Centers need a clear answer for each category and a secure environment for the ones the general cluster cannot take. What follows is an operator's orientation, not legal advice. Your research compliance, export control and information security offices make the determinations.
CUI and NIST SP 800-171
Controlled Unclassified Information (CUI) is information that law, regulation or government-wide policy requires to be safeguarded or disseminated under controls, but that is not classified. The program was established by Executive Order 13556, is overseen by NARA's Information Security Oversight Office, and is implemented through 32 CFR Part 2002.[27] For non-federal systems such as university clusters, the baseline is NIST SP 800-171. Revision 3 was published in May 2024 and supersedes Revision 2. Its requirement families include access control, audit and accountability, configuration management, identification and authentication, incident response, media protection, system and communications protection, system and information integrity, and supply chain risk management, among others.[28]
The revision number your contract names matters. The DoD's CMMC program rule incorporates NIST SP 800-171 Revision 2 by reference for Level 2, and Level 3 adds selected requirements from SP 800-172.[29] So a DoD-sponsored project may be assessed against Rev. 2, while other agencies or newer agreements may point at Rev. 3. Build your enclave to satisfy both, and map controls explicitly.
CMMC Status for Research
CMMC has two rules. The program rule (32 CFR Part 170) took effect December 16, 2024, and the DFARS acquisition rule that puts CMMC requirements into contracts took effect November 10, 2025.[30] Implementation has four phases. Phase 1 began on the DFARS rule's effective date, with Level 1 and Level 2 self-assessments as conditions of award. Phase 2 begins one calendar year later, adding Level 2 third-party (C3PAO) certification for applicable contracts. Phases 3 and 4 follow at one-year intervals, with Phase 4 applying CMMC to all applicable DoD solicitations and contracts, including option periods.[31] By that schedule, Phase 2 starts on November 10, 2026.
For universities the critical point is fundamental research. In the DFARS rule's response to comments, DoD states that fundamental research, as defined in NSDD 189, is published and broadly shared and so cannot be safeguarded as either FCI or CUI. If fundamental research has the potential to become CUI, though, CMMC requirements apply once the data becomes CUI. The rule also recognizes an enclave approach, where the contractor defines the systems used for a DoD contract and the assessment scope in advance.[30] That is exactly why most research universities build a scoped CUI enclave instead of trying to certify the whole campus.
HIPAA and NIH Controlled-Access Data
The HIPAA Security Rule sets national standards for protecting electronic protected health information created, received, used or maintained by a covered entity or business associate. It requires administrative, physical and technical safeguards for confidentiality, integrity and availability, and is codified at 45 CFR Part 160 and Subparts A and C of Part 164.[32] University clusters usually take HIPAA data under a documented institutional arrangement with the health system, with security controls, access reviews and audit logging that the privacy office has approved.
NIH genomic data is the category most likely to surprise a general-purpose center. Since January 25, 2025, users of data from NIH Controlled-Access Data Repositories under the Genomic Data Sharing Policy are expected to secure it according to the NIH Security Best Practices for Users of Controlled-Access Data. Those practices are built on NIST SP 800-171, and users who analyze data on third-party systems or cloud providers must attest that the system complies (ISO/IEC 27001/27002 is accepted as an equivalent).[33] Effective February 25, 2026, the expectation extends to data from all NIH controlled-access repositories under new or renewed agreements.[34] In practice, many dbGaP-style projects now need the same enclave as CUI.
Export Controls Basics
Export controls (EAR and ITAR) regulate not just shipping items abroad but releasing controlled technology or technical data to foreign persons in the United States. Such a release is a "deemed export" to the person's home country.[35] Most university work avoids this through the fundamental research exclusion. Under 15 CFR 734.8, technology or software that arises during or results from fundamental research and is intended to be published is not subject to the EAR. Once researchers accept publication restrictions or decide to keep results proprietary, that protection is lost (with narrow exceptions for brief patent and sponsor-proprietary reviews).[36] For the operator, this means three things. Your export control office, not the systems team, decides whether a project is controlled. Controlled projects need access controls based on citizenship or authorization, which a shared cluster cannot enforce. And input data such as licensed engineering models may be controlled even when the research itself is fundamental.
Secure Enclaves for Research
The durable pattern is a separate, scoped secure environment rather than retrofitting the main cluster. Examples of how centers present this: the University of Michigan's Armis2 is an HPC cluster described as suitable for export-controlled and HIPAA-regulated data, while making clear that users remain responsible for the security and compliance of their sensitive code and data.[37] NIST now publishes HPC-specific guidance for building such systems. SP 800-223 (February 2024) gives an HPC reference architecture, threat analysis and security posture,[38] and SP 800-234 (May 2026) is an HPC Security Overlay that starts from the SP 800-53B moderate baseline and tailors 60 SP 800-53 controls with HPC-specific guidance.[39] Trusted CI, which works with NSF cyberinfrastructure operators on their cybersecurity programs, runs a Regulated Research Community of Practice with monthly webinars.[40]
Features a research enclave should plan for:
- Separate login and data transfer paths with MFA, and no direct internet egress from compute.
- Project-based isolation (per-project groups, storage and ideally partitions or reservations), so data from one agreement never shares a filesystem namespace with another.
- Centralized logging with retention that matches your strictest agreement, plus regular access reviews.
- A system security plan and control mapping that covers NIST 800-171 Rev. 2 and Rev. 3, HIPAA and NIH requirements in one matrix.
- GPUs inside the enclave. Researchers will not accept a secure environment that cannot train models, and if the enclave has no GPUs, they will find riskier alternatives.
Field note: Price the secure enclave honestly and fund the compliance staff time centrally if you can. If a CUI or dbGaP project costs a PI several times what the open cluster costs, they will try to argue their data out of scope, and your security office will be the one saying no in a hallway.
Cost Models and Chargeback
Building a Rate
Whether you charge or not, you need to know your fully loaded cost per unit of service. Georgia Tech's approach is a reasonable template: sum hardware (the portion the center funds), effort-weighted salaries, software licenses and facility costs, divide over the service life, and account for planned maintenance downtime and historical utilization.[3] Set GPU rates per GPU-hour on the GPU type, not per node, and decide deliberately how CPU and memory bundled with a GPU are charged. Otherwise a one-GPU job that takes every core on the node will be billed for one-eighth of it. If the rate is charged to federal awards, the 2 CFR 200.468 rules apply: usage-based, non-discriminatory, aggregate cost recovery, and review at least every two years.[4]
Showback Before Chargeback
Even under a fully subsidized model, send PIs a regular usage
statement. Georgia Tech provided monthly showback statements during
its Phoenix migration before charges began.[3]
Slurm makes this straightforward. Partition
TRESBillingWeights define the billing TRES that feeds
fairshare and billing limits,[41] and
sreport produces utilization reports by account and
user from the accounting database rollups, for any TRES including
GPUs.[42]
# Monthly GPU-hours by account and user (example dates; TRES name must match AccountingStorageTRES)
sreport -T gres/gpu cluster AccountUtilizationByUser start=2026-09-01 end=2026-10-01 -t hours
# Billing-weighted usage, the same number fairshare uses
sreport -T billing cluster AccountUtilizationByUser start=2026-09-01 end=2026-10-01 -t hours
Open XDMoD adds a web portal on top of scheduler logs, with metrics such as job counts, CPU hours, wait time and wall time broken down by any level of your organizational hierarchy, plus an optional job performance module.[43] Georgia Tech used it for exactly this reporting role.[3]
Capacity Planning for AI Demand
AI demand does not grow on the smooth curve that CPU demand used to follow. It jumps when a new model family makes a technique accessible, when a department hires three ML faculty, or when a course moves its labs onto the cluster. Plan in terms of signals and options, not a single forecast.
Signals to Watch
- Queue wait by partition and job size, especially GPU partitions. Rising median wait for one-GPU jobs means you are short of entry-level capacity. Rising wait for multi-node jobs means you are short of contiguous blocks.
- Pending GPU demand (GPU-hours requested by pending jobs) alongside allocated GPU-hours. Slurm 26.05 exposes GPU allocation statistics through its OpenMetrics endpoints, which makes these series easy to collect.[44]
- Actual GPU utilization of running jobs. Buying more GPUs to serve jobs that use a fraction of the ones they hold is the most expensive mistake in this field (see Observability).
- Pipeline signals: faculty hiring plans, large grant submissions with compute budgets, new degree programs, and consultation intake notes. Ask the research office to flag proposals that mention GPUs.
- Offload volume: how many projects you sent to ACCESS, NAIRR, INCITE or cloud, and why.
Planning Levers
Your levers, roughly from fastest to slowest: policy changes (caps, QOS, sharing via MIG), better efficiency (onboarding, data pipelines), brokering national resources and cloud, condo buy-ins timed with grant cycles, and central procurement. Hardware lead times, and above all facility power and cooling, are the slow constraints (see Power, Cooling and Facilities). NSF's Campus Cyberinfrastructure (CC*) program, which includes a "Computing and the Computing Continuum" area, is one external source for campus compute investments.[45] When you plan GPU purchases, favor configurations that can serve both simulation and AI (double-precision capable parts where your simulation codes need them, enough memory per GPU for current models), and plan the network for multi-node training from day one. Retrofitting a fabric costs far more than building it right (see Interconnects).
Staffing and Skills
The skills mix has shifted. A 2015-era center could run on systems administrators and a couple of consultants. A 2026 center that supports AI needs people who can tune NCCL on a fabric, build containers with the right CUDA stack, run Kubernetes for inference endpoints, write a system security plan, teach PyTorch performance, and talk to a dean about cost recovery. CaRCC's facings model is a practical way to map these roles: researcher-facing (facilitation, consulting, training), data-facing (data management, transfer, workflows), systems-facing (infrastructure and operations), and strategy and policy-facing (leadership, governance, compliance).[23][20]
Recruiting and retention are constant problems, because the same skills command industry salaries. CaRCC's RCD Professionalization working group publishes an HR job family matrix that institutions have customized to classify RCD roles properly. It also ran a 2025 RCD Workforce Survey, with initial results published in the PEARC26 proceedings, which provides data on titles, responsibilities and compensation.[23] Use these when you work with HR on job descriptions and salary bands. A reclassification backed by community data is far easier to win than one argued from anecdotes.
- Grow your own. Student employees and interns trained on real operations work are the best pipeline most centers have.
- Cross-train on the new stack. Give systems staff time to learn Kubernetes and GPU operations, and give facilitators time to learn modern ML frameworks, before the demand arrives.
- Share staff. Joint positions with departments or institutes (an embedded ML facilitator, an RSE funded partly on grants) extend reach and build faculty allies.
- Use the community. Campus Champions, CaRCC tracks and US-RSE provide peer support that no single center can replicate.
Sustainability
Sustainability has two meanings in research computing, and centers need to manage both.
Energy and Environmental Sustainability
Data center energy is now a public policy issue. A DOE-commissioned
LBNL report found that U.S. data centers used about 4.4% of total
U.S. electricity in 2023 (176 TWh), and projected 6.7% to 12% by
2028.[46] University GPU clusters are a small
part of that, but campus sustainability offices and utilities notice
every new megawatt. Practical steps: measure energy per job where
you can, use scheduler power saving for idle nodes (Slurm 26.05
allows SuspendTime per node, so power-save policy can
differ by pool),[44] and set efficiency
expectations in policy. Slurm's GPU energy accounting plugin
(acct_gather_energy/gpu) currently supports only AMD's
RSMI library, so NVIDIA sites usually collect GPU power through
DCGM-based monitoring instead.[47]
Facility-level measures such as liquid cooling and heat reuse are
covered in
Power, Cooling and Facilities.
Financial and Organizational Sustainability
The harder problem is keeping the service funded through refresh cycles. Hardware bought with one-time money becomes an unfunded liability in five years. Guard against that by building refresh into the base budget or the rate, keeping a written service catalog with defined levels, documenting governance (a faculty advisory committee with a real charter), and avoiding single-person dependencies for critical systems. Condo contributions, consumption revenue, national resources and grants should add up to a model that survives the departure of the one champion who got it started.
Measuring Impact
Leadership asks one question in many forms: what did we get for this money? Answer with a mix of operational and research outcome metrics, collected continuously rather than assembled in a panic before budget season.
| Category | Metrics | Where the data comes from |
|---|---|---|
| Usage | Active users, PIs, departments; core-hours and GPU-hours delivered | Slurm accounting, sreport, XDMoD |
| Service quality | Queue wait by partition, ticket response and resolution time, uptime | Scheduler logs, ticket system, monitoring |
| Efficiency | GPU utilization of running jobs, fraction of idle allocated GPUs | DCGM and job-level telemetry |
| Reach | Workshop attendance, new departments, first-time users, national allocations obtained | Registration, account records, ACCESS and NAIRR records |
| Research outcomes | Publications and grants that used the resource, graduates trained | Acknowledgment tracking, grant office data, annual surveys |
The return-on-investment literature supports making this case. The Coalition for Academic Scientific Computation (CASC) has run an ROI working group that surveys member institutions and reports on the value of research computing and data services,[48] and a PEARC 2022 review of historical studies concludes that existing literature shows strong evidence of good return on investment in cyberinfrastructure.[49] Local evidence persuades more than national studies, so ask for an acknowledgment statement in publications, link accounts to grant records with your research office, and survey PIs every year. Georgia Tech showed its post-transition impact directly from scheduler data: larger maximum job sizes, more CPU time consumed per group, and much shorter wait times.[3] Provosts believe numbers like those.
Field note: Keep a running file of research stories: one paragraph, one figure, one named PI, one sentence on what the cluster made possible. When the budget call comes, twelve good stories and a utilization chart will do more than any cost-per-core-hour comparison.
Key Takeaways
- Every campus service mixes condo, consumption, subscription, allocation and cloud models. Design the blend deliberately, include a free entry tier, and follow 2 CFR 200.468 when charging federal awards.
- Treat ACCESS, NAIRR (now moving from pilot to the SDSC-led NAIRR Operations Center), INCITE and ALCC as extensions of local capacity, and get researchers onto them early.
- Facilitation is the service. Use Carpentries lessons including HPC Carpentry's hpc-intro, treat documentation as a product, and build a structured GPU onboarding path with an efficiency check.
- Build a scoped secure enclave with GPUs for CUI, HIPAA, NIH controlled-access and export-controlled work. CMMC Level 2 references NIST SP 800-171 Rev. 2 while Rev. 3 is current, CMMC Phase 2 begins November 10, 2026, and fundamental research stays out of scope until data becomes CUI.
- Know your fully loaded cost per GPU-hour, bill GPUs by weight in Slurm, and send showback statements before any chargeback.
- Plan AI capacity from signals (wait times, pending GPU demand, utilization, hiring and grant pipelines), and remember that power and cooling are the slowest constraint.
- Staff for the new stack using CaRCC's facings and job family resources, and fund refresh and governance so the service outlives its founders.
- Measure usage, service quality, efficiency, reach and research outcomes continuously, and pair the numbers with research stories.
Sources
- Community Cluster Program (Purdue University Rosen Center for Advanced Computing)
- HPCC Access and Recharging Rates (UC Riverside High-Performance Computing Center, 2026)
- Jezghani et al., Phoenix: The Revival of Research Computing and the Launch of the New Cost Model at Georgia Tech (PEARC 2022, ACM)
- 2 CFR 200.468 Specialized service facilities (eCFR)
- CloudBank (NSF-supported cloud access service)
- About ACCESS (NSF ACCESS program)
- ACCESS home page (NSF ACCESS program, 2026)
- ACCESS Allocations: Project Types (NSF ACCESS)
- ACCESS Credit Exchange and Calculator (NSF ACCESS)
- ACCESS Allocations: Prepare Requests (NSF ACCESS)
- National Artificial Intelligence Research Resource (NSF)
- NAIRR Pilot portal (NSF and partners, 2026)
- NSF establishes operations center for the National Artificial Intelligence Research Resource (NSF, September 2026)
- Strengthening America's AI Ecosystem with the Launch of the NSF NAIRR Operations Center (SDSC, September 2026)
- About INCITE (DOE Leadership Computing)
- INCITE program home: 2027 call dates (DOE Leadership Computing, 2026)
- ASCR Leadership Computing Challenge (DOE Office of Science)
- Horizon User Guide (TACC, 2026)
- Campus Champions (2026)
- CaRCC People Network (Campus Research Computing Consortium)
- CaRCC Capabilities Model (Campus Research Computing Consortium)
- US Research Software Engineer Association (US-RSE)
- RCD Professionalization Working Group (Campus Research Computing Consortium)
- The Carpentries
- HPC Carpentry Enters Lesson Program Incubation (The Carpentries, July 2024)
- Using High-Performance Computing Systems, hpc-intro lesson (Carpentries Incubator)
- About Controlled Unclassified Information (National Archives, ISOO)
- NIST SP 800-171 Rev. 3, Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations (NIST, May 2024)
- 32 CFR 170.14 CMMC Model (eCFR)
- DFARS: Assessing Contractor Implementation of Cybersecurity Requirements, final rule, 90 FR 43560 (DoD, September 10, 2025)
- 32 CFR 170.3 Applicability, CMMC implementation phases (eCFR)
- The HIPAA Security Rule (HHS)
- NOT-OD-24-157: Implementation Update for Data Management and Access Practices Under the Genomic Data Sharing Policy (NIH, July 2024)
- Requirements for NIH Controlled-Access Data Repositories and Users (NIH)
- Export Control FAQs (UC Santa Barbara Office of Research)
- 15 CFR 734.8 Technology or software that arises during, or results from, fundamental research (eCFR)
- Armis2 HPC Cluster User Guide (University of Michigan ARC)
- NIST SP 800-223, High-Performance Computing Security: Architecture, Threat Analysis, and Security Posture (NIST, February 2024)
- NIST SP 800-234, High-Performance Computing (HPC) Security Overlay (NIST, May 2026)
- Trusted CI, the NSF Cybersecurity Center of Excellence
- Trackable RESources (TRES) (SchedMD, Slurm 26.05 documentation)
- sreport man page (SchedMD, Slurm 26.05 documentation)
- Open XDMoD documentation (University at Buffalo CCR)
- Release Notes for Slurm 26.05 (SchedMD, 2026)
- Campus Cyberinfrastructure (CC*) program (NSF)
- DOE Releases New Report Evaluating Increase in Electricity Demand from Data Centers (U.S. DOE, December 2024)
- slurm.conf man page (SchedMD, Slurm 26.05 documentation)
- Return on Investment Working Group (Coalition for Academic Scientific Computation)
- Stewart et al., Return on Investment in Research Cyberinfrastructure: State of the Art (PEARC 2022)