Installation_
Site Readiness Before the GPUs Arrive: Power, Cooling, Floor, Pathways
A field engineer’s checklist for verifying power, cooling, floor loading, and cable pathways before GPU racks arrive, preventing costly delays in AI data-center deployments.
Key facts
- NVL72 racks can require up to 140 kW each, demanding dedicated busway or hardwired feeds with redundant PDUs.
- Liquid cooling loops must be pressure-tested to the OEM spec (typically 1.5× operating pressure) and flushed to a particle count below the OEM threshold (usually <100 particles/mL).
- Floor loading for a fully populated NVL72 rack exceeds 2,500 kg, requiring a reinforced raised floor or slab with point-load capacity verified per TIA-942.
- MPO trunk cables for scale-out networking must be factory-terminated and polished; field work is limited to patching, routing, cleaning, and testing with an OTDR.
- Copper NVLink spine inside the rack is pre-terminated; site readiness focuses on fiber pathways for InfiniBand/Ethernet between racks and leaf switches.
- Thermal runaway risk from blocked airflow or undersized cooling loops is the #1 cause of GPU throttling in the field, per Leviathan Systems field data.
- A calibrated MPO continuity tester and inspection scope are mandatory for certifying every fiber link before rack power-on.
Power: Capacity, Redundancy, and Termination Verification
Before a single GPU rack lands, the electrical infrastructure must be fully commissioned. For NVL72-class racks drawing up to 140 kW each, the facility must provide dedicated busway or hardwired feeds with redundant PDUs rated for the full load. Verify that each PDU’s breaker rating matches the rack’s nameplate current, and that the upstream UPS and generator can handle the aggregate load of all planned racks without exceeding 80% capacity—this is a standard data-center design rule, not a suggestion.
Termination points must be labeled per the site’s cable management standard (e.g., TIA-606-B) and tested for phase rotation, voltage stability, and ground continuity. Use a calibrated power quality analyzer to confirm voltage stays within ±5% of nominal under a simulated load bank test. Skipping this step risks tripping breakers or damaging GPU power supplies during initial power-on, which can delay the entire build by weeks while replacements are sourced. Ensure the load bank test exercises each phase to within 10% balance to prevent neutral overload.
Cooling: Liquid Loop Preparation and Pressure Testing
Liquid cooling is non-negotiable for NVL72 racks; air cooling alone cannot handle the heat density. The facility’s CDU (coolant distribution unit) must be installed, filled with the specified coolant (typically a propylene-glycol mix), and pressure-tested to 1.5× the operating pressure per the OEM spec—commonly around 6–8 bar. Hold the pressure for 24 hours with less than 0.1 bar drop; any leak requires immediate remediation.
After pressure testing, flush the loop through a 100-micron filter until particle count drops below the OEM threshold (typically <100 particles per milliliter). This prevents debris from clogging cold plates or microchannel heat exchangers. Also verify flow rate and temperature differential across the CDU: at full load, expect a delta T of 10–15°C between supply and return. If the facility’s chilled water loop cannot maintain that delta, the GPUs will throttle. Leviathan Systems has seen builds stall for weeks because the cooling loop was undersized—test it before racks arrive.
Floor Loading and Seismic Bracing
A fully populated NVL72 rack weighs over 2,500 kg, concentrated on a footprint of roughly 0.6 m × 1.2 m. The raised floor or slab must be verified for point-load capacity per TIA-942 guidelines—typically a minimum of 2,000 kg per pedestal for raised floors, or a slab thickness of at least 200 mm with reinforcement. Use a load test with a calibrated hydraulic jack and dial indicator: apply 125% of the expected load and confirm deflection stays within the OEM spec (commonly under 2 mm). For dynamic loads from rolling racks, use a load spreader plate to distribute weight evenly.
Seismic bracing is mandatory in seismic zones (e.g., California, Japan) and recommended elsewhere. Anchor each rack to the floor with bolts rated for the rack’s weight and the region’s seismic acceleration coefficient. For raised floors, use seismic-rated pedestals and cross-bracing. Failure to brace properly can cause racks to shift during a seismic event, severing power and fiber connections—a catastrophic failure mode that is entirely preventable.
Cable Pathways: Fiber and Copper Separation
Scale-out networking (InfiniBand or Ethernet) uses MPO trunk cables between racks and leaf switches. These factory-terminated fibers must be routed in dedicated pathways—separate from power cables by at least 300 mm to avoid EMI, per TIA-569. Use ladder rack or cable tray with a minimum bend radius of 10× the cable diameter for fiber (typically 30 mm for a 3 mm cable). Never exceed the bend radius; micro-bends cause insertion loss that degrades signal integrity. Install bend-radius-compliant guides at every turn.
Copper NVLink spine cables inside the rack are pre-terminated and routed by the rack integrator. Site work focuses on fiber patching: label every MPO connector per the cabling standard, clean it with a one-click cleaner, and inspect with a 200× microscope to confirm end-face quality per IEC 61300-3-35. Then test with an OTDR to verify loss below the OEM threshold (typically 0.5 dB per mated pair). A single dirty or damaged connector can cause link errors that are hard to diagnose after power-on.
Common Failure Modes: What Goes Wrong in the Field
The most frequent failure mode is inadequate cooling loop capacity—the CDU cannot maintain the required delta T under full load, causing GPU throttling. This is often discovered only after racks are powered on, leading to a frantic retrofit. Catch it by running a full-load heat rejection test on the CDU before racks arrive. Simulate the heat load using resistive heaters or a dummy loop; if the delta T drops below 10°C, the loop is undersized.
Second is power imbalance: one phase of a 3-phase feed is overloaded because rack power draw is not evenly distributed across phases. Use a phase rotation meter and load bank to verify balance within 10% before connecting racks. Third is fiber contamination: a single dirty MPO ferrule can cause link flapping that takes hours to isolate. Always inspect and clean every connector before mating. Fourth is floor deflection: a raised floor that passes a static load test may still deflect under dynamic loading from rolling racks—use a load spreader plate and verify deflection limits. Fifth is incorrect coolant mixture: too little glycol reduces heat transfer, too much increases viscosity and pump load—test the mixture per OEM spec.
Commissioning Sequence: Power, Cooling, Network, Compute
The correct order is: power first, then cooling, then network, then compute. Power on the PDUs and verify voltage and phase balance. Then start the CDU and confirm flow rate, pressure, and delta T. Next, bring up the leaf switches and verify fiber links with an OTDR, ensuring all mated pairs are below the loss threshold. Only then power on the compute racks.
This sequence prevents a scenario where GPUs power on but immediately throttle due to no cooling, or where a network issue is masked by a power fault. Each step must be signed off by a qualified engineer before proceeding. Leviathan Systems uses a standardized checklist for each rack, with tolerances for every parameter—deviations halt the build until resolved.
Standards referenced: TIA-942 (Data Center Infrastructure Standards) · TIA-606-B (Cable Administration Standard) · TIA-569 (Pathways and Spaces Standard) · IEC 61300-3-35 (Fiber Optic Connector End-Face Inspection) · ISO 14644-1 (Cleanroom Standards for particle count in cooling loops)
Frequently asked_
How do I verify floor loading capacity for a 2,500 kg rack?
Use a load test with a calibrated hydraulic jack and dial indicator at the rack’s footprint. Apply 125% of the expected load (3,125 kg) and measure deflection—common practice is to stay under 2 mm, but always refer to the OEM spec. For raised floors, also check pedestal spacing and verify that the floor tiles are rated for the point load per TIA-942. If the floor fails, reinforce with additional pedestals or a load-spreading plate.
What is the minimum bend radius for MPO trunk cables in cable trays?
The minimum bend radius for a typical 3 mm diameter MPO trunk cable is 30 mm (10× the cable diameter). Never exceed this; micro-bends cause insertion loss that degrades signal integrity. Use bend-radius-compliant guides at every turn in the tray. For armored or hybrid cables, check the OEM spec—some require a larger radius. Always verify with the cable manufacturer’s datasheet.
How do I test the cooling loop before connecting racks?
Pressure-test the loop to 1.5× the operating pressure (per OEM spec, typically 6–8 bar) and hold for 24 hours with less than 0.1 bar drop. Then flush through a 100-micron filter and measure particle count—must be below the OEM threshold (usually <100 particles/mL). Finally, run the CDU at full flow and verify the delta T between supply and return is within 10–15°C under a simulated heat load. Also confirm the coolant mixture ratio matches the OEM specification.
What is the most common cause of GPU throttling in the field?
Inadequate cooling loop capacity—the CDU cannot maintain the required temperature differential under full load, causing the GPUs to throttle to protect themselves. This is often due to undersized chilled water supply, clogged filters, or incorrect coolant mixture. Always run a full-load heat rejection test on the CDU before racks arrive. Leviathan Systems has seen this cause weeks of delays in large deployments.
Can I field-terminate MPO connectors for GPU networking?
No. MPO trunk cables for scale-out networking (InfiniBand/Ethernet) are factory-terminated and polished. Field work is limited to patching, routing, cleaning, inspection, and testing with an OTDR. Field-terminating MPO ferrules is not recommended because it introduces high insertion loss and reliability issues. Always use pre-terminated cables from a reputable vendor and test every link before rack power-on.