Power, Cooling and Facilities

For most of the history of research computing, the machine room was the part of the project you could take for granted. You asked facilities for a few more 208V circuits, checked that the CRAC units had headroom, and got on with building the cluster. That era is over. A single current-generation GPU rack draws more power than an entire row of the CPU clusters described in the 2019 edition of this manual, it cannot be cooled with air, it weighs about as much as a small car, and the electrical gear that feeds it can take longer to arrive than the computers. This chapter walks through what changed, what the numbers actually are (from vendor documentation and industry surveys, not rumor), and how a university should plan for its first liquid-cooled GPU racks. For the hardware itself see Accelerators; for the cloud alternative see Cloud and Hybrid AI/HPC.

Rack Density: Then and Now

The single most important number in facilities planning is kilowatts per rack. It drives circuit sizing, cooling technology, floor loading, UPS capacity and, ultimately, whether a given room can host the system at all.

The typical enterprise rack

Uptime Institute has surveyed data center operators every year for more than a decade, and the picture for the broad industry is still modest. The 2024 survey found that 4 kW to 6 kW racks remained the most commonly deployed density, that 7 kW to 9 kW racks had become notably more common, and that the average of typical rack densities across respondents was 8 kW (7.1 kW if 11 sites reporting more than 50 kW were excluded as outliers). The majority of facilities had no racks above 30 kW at all.[1] The 2025 survey reported continued increases, with greater adoption of racks in the 10 kW to 30 kW range, while noting that few facilities exceed 30 kW and that extreme densities are still rare.[2] Microsoft describes its general-purpose cloud compute and storage racks the same way: usually below 20 kW.[3]

That is the world most campus machine rooms were designed for: raised floor or slab, perforated tiles or in-row coolers, and single- or three-phase whips sized for 5 kW to 15 kW per cabinet.

The rack-scale GPU system

NVIDIA's rack-scale Blackwell systems (GB200 NVL72 and GB300 NVL72) put 72 GPUs and 36 Grace CPUs in a single 48U rack, with 18 compute trays, 9 NVLink switch trays, power shelves and a busbar, all liquid cooled.[4] The vendor documentation for these racks gives the following power figures:

Put those side by side with the Uptime numbers and the change is stark: one GPU rack at roughly 132 kW to 135 kW nominal is on the order of 16 times the 8 kW survey average, and it can briefly exceed that nominal figure. NVIDIA itself frames the shift as data centers that "once operated at 20 kW per rack" moving to hyperscale facilities that "can support over 135 kW per rack".[8]

Where density is heading

The trajectory is still upward. Google has stated publicly that machine learning will require more than 500 kW per IT rack before 2030.[9] NVIDIA's roadmap describes 800 VDC power infrastructure intended to support 1 MW IT racks starting in 2027, timed with its Kyber rack generation.[10] The successor to GB300, Vera Rubin NVL72, is described by NVIDIA as in full production, is built on the third-generation MGX NVL72 rack design, and is specified for 45 C liquid inlet temperature.[11] NVIDIA's own material on power provisioning for these racks states that its dynamic power software reduced provisioned rack power from 136 kW to 101 kW for Vera Rubin NVL72 on a tested inference workload (a vendor claim about its own software, but a useful indication of the provisioning figure).[12] Treat any per-rack figure for Rubin Ultra or Kyber as roadmap until a vendor publishes a datasheet.

Field note: Always ask for three numbers per rack: nominal (TDP), peak electrical design power, and what the vendor recommends you provision. HPE's GB200 NVL72 guidance is a good example of why: 132 kW nominal, but provision the busway for about 192 kW.[6] Designing to nameplate average is how you trip breakers during a synchronized training step.

Power Delivery: From the Utility to the GPU

Power reaches a GPU through a chain: utility service and substation transformers, switchgear, UPS, distribution (PDUs or overhead busway), the rack power shelf, the in-rack busbar and finally the DC-DC converters on the board. Each hop has changed.

Facility distribution: busways and three-phase AC

High-density rooms in North America commonly distribute three-phase power at 415/240 V or 480/277 V wye, often fed to racks from overhead busway with tap-off boxes rather than under-floor whips. NVIDIA's own description of the status quo is "traditional 415 or 480 VAC three-phase systems".[13] The Lenovo GB300 NVL72 documentation shows what the rack actually expects: each 33 kW power shelf holds six 5.5 kW hot-swappable PSUs with a wide input range of 200-277 VAC (346-480 VAC wye, 5 wires), and each shelf is fed by one 60 A IEC 60309 power whip.[7] HPE's GB200 documentation likewise specifies an IEC 60309 5-wire connector per shelf.[6] With six to eight shelves, that is six to eight 60 A three-phase whips per rack, which is why overhead busway has become the default.

In-rack DC: 48 V, 50 V and 54 V busbars

Inside the rack, the power shelves rectify AC to low-voltage DC and drive a vertical copper busbar that the trays clip onto. Google championed 48 VDC in-rack distribution almost a decade ago, and describes that architecture as having scaled from 10 kW to 100 kW IT racks.[9] NVIDIA describes the GB200 and GB300 NVL72 as using 54 VDC distribution with "up to eight power shelves".[10] Lenovo's GB300 documentation specifies 50 V nominal output (47.5 V to 51.5 V input range at the tray) on an MGX-compliant busbar rated at 1400 A.[7] The labels 48 V, 50 V and 54 V all refer to this same class of low-voltage in-rack bus; check the specific vendor document for the exact nominal value.

The physics problem is current. At roughly 50 V, every 50 kW is about 1000 A. NVIDIA notes that at MW scale a 54 VDC rack would need up to 200 kg of copper busbar and that power shelves would consume up to 64 U, leaving no room for compute.[10]

Higher-voltage DC roadmaps: 400 V and 800 V

The industry answer is to move conversion out of the IT rack and distribute DC at much higher voltage. Two related efforts are worth knowing:

• Mt Diablo (+/-400 VDC). Google, Meta and Microsoft are collaborating through the Open Compute Project on a disaggregated "sidecar" power rack that converts AC to DC outside the IT rack, with +/-400 VDC delivery that Google says can support up to 1 MW per rack. The choice of 400 VDC is explicitly meant to reuse the electric-vehicle supply chain.[9] Microsoft notes the first disaggregated power racks use the existing 48 VDC ecosystem, with the larger gains coming at 400 VDC, and that moving power out of the server rack enables up to 35% more AI accelerators per rack.[3]
• NVIDIA 800 VDC. NVIDIA's architecture converts 13.8 kV AC grid power to 800 VDC at the facility perimeter, runs 800 V busways to the row, and feeds racks with a two-conductor 800 V input and in-rack DC-DC conversion. NVIDIA claims 85% more power through the same conductor size and 45% less copper than 415 VAC, and up to 5% better end-to-end efficiency than current 54 V systems. Full-scale production is planned to coincide with Kyber racks in 2027.[10] NVIDIA reported that more than 20 companies are joining its 800 VDC partner ecosystem.[13]

Both are roadmap for most sites. NVIDIA itself notes that facility-level DC introduces new challenges in safety, standards and workforce training.[10] For a university buying GB300 or Vera Rubin class racks in 2026, the practical design is still three-phase AC to MGX power shelves. What you can do now is avoid decisions that make a later DC retrofit harder: leave space in electrical rooms, prefer overhead distribution, and ask your electrical engineer about the path to DC busway.

Power swings and grid interaction

Large training jobs run thousands of GPUs in lockstep, so the whole cluster's power draw rises and falls together at each step. NVIDIA, Microsoft and OpenAI have published on this problem, and NVIDIA notes that when demand ramps suddenly it can take one minute to 90 minutes for generation resources to respond. The GB300 NVL72 power shelves add electrolytic capacitor energy storage (about 65 joules per GPU) plus software ramp limits and a "power burner" mode for ramp-down, and NVIDIA reports these reduced peak grid demand by up to 30% on a Megatron training run.[14] NVIDIA says the Vera Rubin NVL72 rack adds a liquid-cooled busbar and 20x more energy storage.[13] Tell your utility and your electrical engineer that the load is a synchronized, fast-changing one, not a flat data center load.

Cooling: Why Air Ran Out

Air cooling and its limits

ASHRAE TC 9.9's thermal guidelines define the air environment IT equipment is built to. The recommended inlet range for classes A1 to A4 is 18 C to 27 C; allowable ranges run from 15-32 C (A1) up to 5-45 C (A4). The 2021 edition added a high-density air-cooled class, H1, with a tighter recommended range of 18 C to 22 C (allowable 15-25 C), intended for zones that host dense air-cooled products.[15] That H1 class exists precisely because dense air-cooled servers need colder air to stay within limits.

Well-run air cooling still matters for the 10% or so of heat that liquid-cooled racks reject to air, and for storage, network and management racks. NVIDIA's DGX SuperPOD design guide is a good checklist: hot/cold aisle orientation, aisles at least 36 inches wide (48 inches recommended for cold aisles so server lifts fit), containment, blanking panels and brush grommets, filter and tile audits, and a CFD model of the room before changes.[16] The same guide works an example where 358 kW of heat load in a constrained area had only 260 kW of cooling, resolved by spacing racks to borrow cooling capacity from adjacent footprints, at the cost of floor space and longer cables.[16]

Rear-door heat exchangers

A rear-door heat exchanger (RDHx) replaces the back door of a rack with a water coil (and often fans), so server exhaust is cooled before it enters the room. Vendors rate active doors in the tens of kilowatts; for example, Motivair's ChilledDoor literature states capacity up to 75 kW per rack.[17] RDHx is an excellent retrofit tool for dense air-cooled CPU or PCIe GPU racks, and for capturing the residual air-side heat of liquid-cooled racks. It is not a substitute for direct liquid cooling on a 130 kW rack-scale system.

Direct-to-chip liquid cooling

Direct-to-chip (DLC, also called cold-plate cooling) runs coolant through cold plates bolted to the GPUs, CPUs and switch ASICs. For GB300 NVL72, Lenovo documents that the CPU, GPU and HBM are liquid cooled while front I/O, the power distribution board and M.2 remain air cooled, giving a rack-level split of about 90% of heat to liquid and 10% to air.[7] The liquid system consists of a coolant distribution unit, rear manifolds, quick disconnects and cold plates. Lenovo supports deionized water (recommended) or PG25 (25% propylene glycol) as coolant, and the rack ships dry.[7]

Two loops matter, and the vocabulary is worth getting right because it appears in every contract:

• Technology cooling system (TCS): the secondary loop that moves heat between the racks, cold plates and the CDU. It carries treated coolant at controlled quality.[12]
• Facility water system (FWS): the primary loop that takes heat from the CDU to chillers, dry coolers, cooling towers or a heat-reuse system.[12]

Coolant distribution units

The CDU is a liquid-to-liquid heat exchanger with pumps, filtration and controls that isolates the TCS from facility water. Form factors in current vendor catalogs include:

• In-rack CDU: Supermicro lists up to 250 kW capacity with N+1 redundant pumps.[18]
• In-row CDU: Supermicro lists up to 1.8 MW supporting up to 8 racks[18]; HPE lists an in-row CDU rated for 1.3 MW supporting a maximum of 8 racks, with overhead or underfloor secondary loop options.[6]
• Liquid-to-air (L2A) sidecar: Supermicro lists up to 200 kW with no facility water required, rejecting heat to room air.[18] This is a bridge for rooms without facility water, but the heat still lands on your room air system.

Field note: Budget for CDU redundancy the way you budget for UPS redundancy. An in-row CDU serving eight racks is a single point of failure for roughly a megawatt of GPUs. Ask how the CDU behaves on loss of facility water, loss of one pump, and loss of utility power, and how long the TCS can ride through before GPUs throttle or trip.

Facility water temperatures: ASHRAE W classes

ASHRAE classifies liquid-cooled equipment by the facility water supply temperature it can accept. The classes were renamed so the upper limit appears in the name; all share a 2 C minimum.[19]

Sources for table: ASHRAE 2021 Thermal Guidelines reference card, Table 3.1[15]; ASHRAE AI Data Center Energy Performance Framework[19].

ASHRAE describes W17/W27 facilities as typically chiller-based, W32/W40 as typically operated without chillers in most locations, and W45/W+ as typically chiller-free to save energy and capital, though some locations are unsuitable for dry coolers.[15]

Current GPU racks sit at the warm end. Lenovo states allowable water inlet as high as 45 C for GB300 NVL72, and lists the NVLink switch tray as ASHRAE W45 compliant (2 C to 50 C operating water).[7] NVIDIA specifies 45 C inlet for Vera Rubin NVL72 and argues that warmer inlet lets facilities use free cooling more often and rely less on chillers or evaporative coolers.[11][12] That is good news for universities: a W32 to W45 loop can often be served by dry coolers or towers most of the year, and the return water is warm enough to be useful for heating.

Immersion cooling

Immersion submerges whole servers in a dielectric fluid, either single-phase (the fluid stays liquid and is pumped through a heat exchanger) or two-phase (the fluid boils on hot components and condenses on a coil). The Open Compute Project's Immersion Requirements document defines terminology, fluid property tests, and customer requirements, including spill containment sized to at least 100% of the largest container, spill absorption materials, and fluid safety documentation (MSDS and TDS) in the room.[20] Immersion works and has real deployments, but for 2026-era rack-scale GPU systems the vendor-supported path is direct-to-chip in MGX racks. Unless your vendor explicitly supports and warrants a specific GPU platform in a specific immersion tank, treat immersion as a research project, not a procurement option.

PUE, WUE and Heat Reuse

Power usage effectiveness

PUE is total facility energy divided by IT equipment energy over a year (standardized as ISO/IEC 30134-2). A PUE of 1.5 means half a watt of overhead (cooling, UPS losses, lighting) for every watt that reaches IT gear. Reference points from primary sources:

• Industry average: Uptime reported 1.56 in its 2024 survey[1] and "little change for the sixth consecutive year" in 2025.[2] Uptime also notes that many recent builds consistently achieve 1.3 or better.[1]
• Hyperscale: Google reports a 2025 fleet-wide trailing twelve-month PUE of 1.09.[21]
• Warm-water research facility: the National Laboratory of the Rockies (formerly NREL) HPC data center reports an annualized average PUE of 1.036 using warm-water liquid cooling, with no mechanical chillers.[22]

Two cautions. First, PUE says nothing about whether the IT load does useful work; an idle GPU cluster with PUE 1.1 is still waste (see Observability, Performance and Efficiency). Second, moving fans from the facility into the servers (or vice versa) shifts energy across the PUE boundary. Liquid cooling removes most server fans, which lowers IT energy and can make PUE look slightly worse even as total energy falls.

Water usage effectiveness

The Green Grid defines WUE as annual site water usage divided by IT equipment energy, in liters per kWh, using the same IT energy denominator as PUE. Site water includes cooling tower evaporation, blowdown and drift, and humidification. A second metric, WUEsource, adds the water consumed off-site to generate the electricity.[23] For scale, Microsoft reports its average data center WUE fell from 2.3 L/kWh in its first-generation sites to 0.27 L/kWh in 2025.[24]

Closed-loop liquid cooling with dry coolers uses very little site water; evaporative towers use much more but save electricity. NVIDIA summarizes the tradeoff: evaporative cooling is far more energy-efficient than chillers but can consume millions of gallons of water per megawatt annually, while dry coolers use no water but depend on ambient temperature and pair best with warm-water IT.[8] In drought-prone regions (much of the University of California system, for example) the water number may matter as much to your administration as the energy number.

Heat reuse

Warm-water liquid cooling turns waste heat into a usable resource. Examples with public documentation:

• NLR ESIF (Colorado): the data center captures and reuses waste heat as the primary heating source for the building's offices and laboratory space.[22]
• CSC Kajaani (Finland), home of LUMI: excess heat is recovered and fed into the City of Kajaani's district heating network, covering the annual heating needs of hundreds of households.[25]

ASHRAE's W-class table explicitly lists district heating as a secondary facility for W32 through W+ classes.[15] For a campus with a central plant and a hot-water or steam loop, the questions are practical: what return temperature will the GPU loop deliver, what temperature does the campus heating loop need, is a heat pump required to bridge the gap, and who owns the heat exchanger. Campus energy managers are often enthusiastic partners once they learn the heat is available year-round.

Field note: Heat reuse only pays when there is a year-round heat sink. A building that needs heat only in winter will force you to reject most of the heat anyway. Bring the campus energy office in during design, not after the racks arrive.

Floor Loading, Weight and Space

HPE lists the GB200 NVL72 rack at approximately 3,245 lbs fully loaded, with a footprint of 600 mm wide by 1,068 mm deep.[6] Supermicro gives the same 600 x 1068 mm footprint and a height of 2,236 mm for its 48U GB300 NVL72 rack.[18]

Worked example (arithmetic from the HPE figures above): the footprint is 0.600 m x 1.068 m = 0.64 m2 (about 6.9 ft2). 3,245 lbs over 6.9 ft2 is roughly 470 lbs/ft2 (about 2,300 kg/m2) of static load directly under the rack, before counting coolant in manifolds and overhead busway. Many raised floors and older upper-story rooms were not designed for that concentrated load. Actions:

• Get a structural engineer to evaluate concentrated and rolling loads on the actual path from the loading dock to the final position, including elevators, ramps and raised-floor panels.
• Consider slab-on-grade placement with overhead power and liquid distribution; Lenovo offers top-feed or bottom-feed manifold options, and HPE supports overhead (top feed) or underfloor (bottom feed) secondary loops.[7][6]
• Check door heights: at 2,236 mm plus shipping pallet, a 48U rack will not fit through a standard 7-foot (2,134 mm) door.
• Plan leak containment. NVIDIA's DGX GB rack documentation includes leak detection as a rack subsystem, and Lenovo lists a rack leak tray per NVIDIA reference.[4][7]

UPS and Generator Sizing

Research computing has traditionally run compute nodes on utility power with UPS and generator backup reserved for storage, networking and management. That remains a sensible default for GPU clusters: a training job should checkpoint and survive a node loss (see Training at Scale), while a parallel file system that loses power mid-write can take days to recover. The considerations that change with GPU racks:

• Size to peak, not nominal. UPS and generator capacity for any protected GPU load must cover peak electrical design power (for example, HPE's roughly 1.5x TDP figure for GB200 NVL72).[6]
• Load steps. Synchronized training produces fast load swings. Generators and UPS inverters must handle the step changes without instability; ask the vendor for step-load test data, and note that rack-level energy storage and ramp limiting (as in GB300 NVL72 power shelves) exists partly for this reason.[14]
• Cooling must ride through too. If compute stays up on UPS but CDU pumps or facility pumps do not, the GPUs will overheat in seconds to minutes. Put CDU pumps and controls on the same protected power as anything they cool.
• Redundancy topology. The DGX SuperPOD guide points out that with N+1 power provisioning, N equals multiple circuits, and cooling capacity must be aligned to N rather than to a single circuit.[16]
• Right-sizing matters for efficiency. Uptime found that one in four data centers in its 2024 survey is typically under 40% utilized by available UPS capacity, which undermines the energy performance of power distribution.[1]

Campus Data Center, Colocation or Cloud

Before designing a room, decide whether you should own the room at all.

Uptime notes that colocation capacity is scarce in many locations and equipment lead times remain lengthy[1], and its 2025 supply chain survey found that many operators, colocation providers included, continue to face delays on capital projects.[26] Do not assume a colocation provider can take 130 kW liquid-cooled racks next quarter; ask for the specific hall, the facility water temperature and capacity, and the date. Many universities end up with a hybrid: a modest owned liquid-cooled footprint for sustained workloads, cloud for bursts and early access (see Cloud and Hybrid AI/HPC), and national resources for the largest runs.

Lead Times and Procurement

The computers are often not the long pole. The electrical and mechanical plant is.

• Utility transformers. A May 2026 report from the National Laboratory of the Rockies, citing GAO and DOE, states that lead times for typical large power transformers have lengthened from 12-18 months to 30-36 months from purchase order to receipt, and up to 60 months for extra-high-voltage units.[27] If your project needs a new substation or service upgrade, this can dominate the schedule.
• Critical power and cooling equipment. Uptime's 2025 Data Center Spending and Supply Chain Survey found fewer disruptions than at any point in the prior five years, but delays persist; UPS systems, switchgear, cooling units, engine generators and PDUs remain the most affected categories, and manufacturing and equipment delays are the most-cited driver of capital project delays.[26]
• Rack-scale systems. GB200/GB300 NVL72 racks are integrated and tested at the factory and delivered as complete racks, which shortens on-site build but ties delivery to vendor allocation.

Practical procurement advice:

• Order long-lead facility equipment (switchgear, transformers, CDUs, dry coolers) before the compute RFP closes, sized for the generation after the one you are buying.
• Write facility acceptance criteria into the compute contract: supported water temperature range, coolant chemistry, flow and pressure-drop per rack, leak detection integration, and who commissions the TCS.
• Require the vendor's facility requirements document (every major OEM has one) as a contract deliverable, and hold a joint review with campus facilities and your mechanical and electrical engineers.
• Budget for filling, flushing and water-quality testing. Lenovo offers a service kit with a flow meter, bleed hose, pressure gauge and vent valve for on-site setup.[7]

Field note: The most common schedule failure I see is not late GPUs, it is a finished room waiting on a switchboard, or racks sitting on the dock waiting for the CDU commissioning crew. Put the facilities milestones on the same project plan as the compute delivery, with named owners.

Sustainability and the Green500

The Green500 ranks TOP500 systems by HPL performance per watt. The June 2026 list is instructive for anyone arguing that efficiency and performance are at odds:[28]

• The top three are identical-architecture BullSequana XH3000 systems using NVIDIA GH200 Grace Hopper Superchips with quad-rail InfiniBand NDR200: KAIROS (CALMIP / University of Toulouse) at 73.28 GFlops/W, ROMEO-2025 at 70.91 GFlops/W, and the Levante GPU extension at DKRZ at 69.43 GFlops/W.
• TOP500 notes that efficiency tends to decrease slightly with system size, which gives smaller systems of the same technology an advantage.

On the June 2026 TOP500, El Capitan (No. 2) achieves 60.94 GFlops/W while delivering 1.809 EFlop/s on HPL, and JUPITER Booster (No. 5) uses Eviden's direct liquid-cooled BullSequana XH3000 architecture.[29] The pattern across the top of both lists is the same: liquid-cooled, accelerator-dense systems.

For a campus, sustainability work that actually moves the needle includes: warm-water cooling to eliminate chiller hours, heat reuse, measuring and publishing PUE and WUE for the research computing facility, sourcing renewable electricity, and, most of all, keeping the GPUs busy with useful work. The cheapest kilowatt-hour is the one an idle GPU did not burn.

Planning Checklist: A University's First Liquid-Cooled GPU Racks

Use this as a starting agenda for the first meeting between research computing, facilities, and your engineering consultants. Items are ordered roughly by lead time.

Scope and site

1. Define the target: number of racks, platform family (for example GB300 NVL72 or Vera Rubin NVL72), and growth for the next two generations.
2. Collect the OEM facility requirements document for the exact configuration, including nominal, peak and recommended-provisioning power per rack.[6][7]
3. Decide campus vs. colocation vs. cloud for each workload class before committing capital.
4. Pick the room. Confirm structural capacity for roughly 3,000+ lb racks (about 470 lbs/ft2 under the footprint in the HPE example), the delivery path, and door heights.

Electrical

1. Request a utility capacity study early; if a new service or transformer is needed, assume multi-year lead time.[27]
2. Design three-phase distribution (415/240 V or 480/277 V) on overhead busway with tap-offs matching the rack's power shelf whips (for example, one 60 A IEC 60309 5-wire whip per 33 kW shelf).[7]
3. Decide redundancy: 6 vs. 8 power shelves per rack (non-redundant vs. redundant in Lenovo's GB300 configurations)[7], and A/B feeds.
4. Decide what rides on UPS and generator. At minimum: storage, network, management, CDU pumps and controls.
5. Tell the utility the load is synchronized and fast-changing; ask about ramp-rate limits.[14]
6. Leave physical space and conduit paths for a future higher-voltage DC distribution retrofit.

Mechanical and water

1. Choose the facility water class. Target W32 to W45 so dry coolers or towers can carry most hours without chillers.[15]
2. Select CDU form factor (in-rack, in-row, or liquid-to-air sidecar) and redundancy; confirm capacity per rack and per CDU.[18][6]
3. Specify TCS coolant (DI water or PG25 per vendor support), filtration, water-quality testing schedule and who owns it.[7]
4. Size room air cooling for the residual air-side heat (about 10% of rack power for GB300 NVL72 per Lenovo) plus all air-cooled racks.[7]
5. Plan leak detection and integrate alarms with your monitoring and building management system.[4]
6. Evaluate heat reuse with the campus energy office.
7. Estimate annual water use for the chosen heat-rejection method and report WUE alongside PUE.[23]

Operations and procurement

1. Put facility acceptance criteria and commissioning responsibilities in the compute contract.
2. Train facilities and research computing staff on CDU operation, quick disconnects, fill and drain, and leak response.
3. Stock onsite spares; Lenovo recommends about 2% of order quantity for compute and switch trays to support next-business-day service targets.[7]
4. Instrument everything: rack power, CDU supply and return temperatures, flow, and GPU telemetry, feeding the same monitoring stack (see Observability, Performance and Efficiency).
5. Write an incident runbook for loss of facility water, loss of CDU, leak detection, and utility outage.

Key Takeaways

• Typical enterprise racks still average around 8 kW; rack-scale GPU systems such as GB200/GB300 NVL72 are rated around 132 kW to 135 kW nominal, with peaks well above that.[1][6][7]
• Provision power for the vendor's peak or recommended figure, not nameplate average, and expect synchronized load swings.
• Three-phase AC to in-rack 48-54 V busbars is today's design; +/-400 VDC and 800 VDC distribution are roadmap items aimed at 2027 and later.[9][10]
• Direct-to-chip liquid cooling with a CDU is mandatory for rack-scale GPU systems; warm facility water (W32 to W45) enables chiller-free operation and heat reuse.[15][11]
• Weight, door heights and floor structure are real constraints: about 3,245 lbs per GB200 NVL72 rack.[6]
• Electrical gear, not GPUs, often sets the schedule; large power transformer lead times have reached 30-36 months.[27]
• Report PUE and WUE, pursue heat reuse, and remember that idle GPUs waste more energy than any cooling inefficiency.

Sources

1. Uptime Institute Global Data Center Survey 2024 (Uptime Institute, 2024)
2. Uptime's 15th Annual Global Data Center Survey Results (Uptime Institute press release, 2025)
3. Mt Diablo - Disaggregated Power Fueling the Next Wave of AI Platforms (Microsoft Azure Infrastructure Blog, 2024)
4. Hardware - NVIDIA DGX GB Rack Scale Systems User Guide (NVIDIA)
5. NVIDIA GB200 NVL72 48U Rack Solution specifications (Supermicro)
6. NVIDIA GB200 NVL72 by HPE QuickSpecs (HPE)
7. Lenovo NVIDIA GB300 NVL72 Rack Scale AI Product Guide LP2357 (Lenovo Press)
8. Chill Factor: NVIDIA Blackwell Platform Boosts Water Efficiency (NVIDIA Blog, 2025)
9. AI infrastructure is hot: new power distribution and liquid cooling infrastructure (Google Cloud Blog, 2025)
10. NVIDIA 800 VDC Architecture Will Power the Next Generation of AI Factories (NVIDIA Technical Blog, 2025)
11. NVIDIA Vera Rubin NVL72 product page and specifications (NVIDIA, 2026)
12. Maximizing AI Factory Performance per Watt with NVIDIA DSX MaxLPS (NVIDIA Technical Blog, 2026)
13. NVIDIA, Partners Drive Next-Gen Efficient Gigawatt AI Factories in Buildup for Vera Rubin (NVIDIA Blog, 2025)
14. How New GB300 NVL72 Features Provide Steady Power for AI (NVIDIA Technical Blog, 2025)
15. 2021 Equipment Thermal Guidelines for Data Processing Environments, reference card (ASHRAE TC 9.9, 2021/2024)
16. Cooling and Airflow Optimization - NVIDIA DGX SuperPOD Data Center Design (NVIDIA)
17. Motivair ChilledDoor Rack Cooling System brochure (Motivair, 2023)
18. SuperServer SRS-GB300-NVL72 specifications (Supermicro)
19. AI Data Center Energy Performance Framework: Introduction and Purpose (ASHRAE)
20. OCP Immersion Requirements Rev. 2 (Open Compute Project)
21. Power usage effectiveness (Google Data Centers)
22. High-Performance Computing Data Center (National Laboratory of the Rockies, 2026)
23. Water Usage Effectiveness (WUE): A Green Grid Data Center Sustainability Metric (The Green Grid, 2011)
24. Inside Microsoft's two-decade push to cut water intensity while scaling for growth (Microsoft, 2026)
25. Energy-efficient Kajaani data center (CSC - IT Center for Science)
26. Supply chains show signs of stability, but delays persist, Field Report 189 (Uptime Institute, 2025)
27. Large Power Transformer Supply Chain Gap Analysis and Domestic Content Strategies for Hydropower Rehabilitation (National Laboratory of the Rockies, 2026)
28. Green500 June 2026 (TOP500, 2026)
29. TOP500 June 2026 (TOP500, 2026)