Platforms_
GB300 NVL72 Deployment: Power, Cooling, and the Cable Plant
A field engineer's guide to the physical-layer demands of deploying a GB300 NVL72 rack, covering power distribution, liquid cooling, and the scale-out cable plant, with emphasis on common failure modes and practical steps to avoid them. Based on field experience from Leviathan Systems and industry best practices.
Key facts
- GB300 NVL72 racks require a dedicated 480V AC 3-phase power feed per rack, with a typical draw of 30–40 kW at full load.
- The rack's internal NVLink spine is copper-based; all MPO/fiber cabling in the rack carries only scale-out network traffic (InfiniBand or Ethernet).
- Liquid cooling for GB300 uses direct-to-chip cold plates on GPUs and CPUs, with a facility-side coolant distribution unit (CDU) supplying a dielectric fluid at an OEM-specified temperature range (typically 25–35°C).
- MPO trunk cables used in the scale-out network are factory-terminated and polished; field work is limited to patching, routing, cleaning, inspection, and testing.
- A single GB300 NVL72 rack can require over 200 MPO connections between compute nodes and the top-of-rack switch for the scale-out fabric.
- Per TIA-568.3-D, the maximum insertion loss for a mated MPO connector pair in a structured cabling channel is typically 0.75 dB for multimode fiber (Grade A connectors).
- Coolant flow rate per rack is typically 10–20 liters per minute, depending on the OEM's thermal design power specification for the GB300 GPU.
Power Distribution: 480V AC and the Rack-Level Busway
Every GB300 NVL72 rack arrives with a factory-installed power shelf that expects a 480V AC 3-phase feed. The facility must provide a dedicated circuit from the main switchgear, typically via a busway or a hard-wired whip, terminated at a rack-mounted power distribution unit (PDU) or directly into the rack's busbar. The PDU must be rated for the rack's full load, which can exceed 30 kW under sustained GPU compute. Do not assume a standard 208V feed will suffice; the GB300's power supply units (PSUs) are designed for 480V to minimize current and copper losses.
The breaker at the panel must be sized per local code and the OEM's specification—commonly a 60A or 100A 3-pole breaker per rack. Always verify the phase rotation matches the rack's internal wiring; a phase reversal can damage PSUs. Use a phase rotation meter on the whip before connecting the rack. The ground conductor must be sized per NEC Article 250, and a bonding jumper is required between the rack frame and the facility ground grid. Document the voltage readings and phase rotation in the commissioning report.
Liquid Cooling: Direct-to-Chip and the CDU Interface
The GB300 NVL72 uses direct-to-chip liquid cooling for the GPUs and CPUs. The rack has two blind-mate fluid couplings on the rear panel: one supply and one return. These connect to the facility's coolant distribution unit (CDU) via dielectric hoses. The CDU must be located within a reasonable distance—typically less than 30 meters—to keep pressure drop within the pump's capability. The coolant is a dielectric fluid, not water, to mitigate electrical risk in case of a leak.
The OEM specifies a supply temperature range, typically between 25°C and 35°C, and a maximum pressure of 3 bar at the rack inlet. Before commissioning, flush the facility loop to remove debris that could clog the rack's microchannel cold plates. Use a particle filter with a mesh size per the OEM spec (commonly 50 microns) at the CDU outlet. After connecting the hoses, perform a pressure hold test at 1.5 times the maximum operating pressure for 30 minutes. A pressure drop greater than 5% indicates a leak. Do not power on the rack until the coolant flow is verified and the CDU reports stable temperature and flow rate.
The Scale-Out Cable Plant: MPO Routing and Testing
The GB300 NVL72's scale-out network—InfiniBand or Ethernet—connects each compute node to the top-of-rack (ToR) switch via MPO trunk cables. These are factory-terminated, 12- or 24-fiber MPO assemblies. Field work is patching, routing, cleaning, inspection, and testing. Route cables in the overhead cable tray or underfloor, maintaining a bend radius no less than 10 times the cable diameter per TIA-568.3-D. Use a cable ladder or mesh basket with a bend radius guide at every turn. Do not pull cables with more than 50 pounds of tension; use a pull cord and a tension meter.
After routing, clean every MPO connector endface with a dry-click cleaner followed by a wet-dry wipe using a lint-free cloth and isopropyl alcohol. Inspect each endface with a 200x or 400x fiber microscope; reject any connector with scratches, pits, or debris per the criteria in IEC 61300-3-35 (e.g., for Grade A, no scratch wider than 0.5 microns for single-mode, no debris larger than 2.5 microns). Test each link with an OLTS or OTDR to verify insertion loss is within the budget—per TIA-568.3-D, typically 0.75 dB per mated pair for multimode. Record the test results for each link in the commissioning report.
Common Failure Modes: What Goes Wrong and How to Catch It
The most frequent failure in the field is a contaminated or damaged MPO connector endface. A single speck of dust can cause 1 dB loss or back-reflection that degrades the link. Always inspect before mating, even if the connector came from the factory with a dust cap. Use a clicker cleaner and isopropyl alcohol wipe, then inspect under a microscope. If the endface is scratched or pitted, replace the cable—field polishing of MPO connectors is not feasible.
The second failure mode is a kinked or crushed fiber trunk cable. This happens when cables are pulled around sharp edges or pinched under rack feet. Use a cable tension meter during pull and inspect the entire run for pinch points after installation. The third failure is a coolant leak at the blind-mate coupling. This is often caused by misalignment during rack installation. Ensure the rack is level and the coupling is fully seated before pressurizing. A fourth failure is power phase imbalance. If the facility's 480V feed has an unbalanced load, the rack's PSUs may trip on undervoltage. Monitor the phase voltages at the PDU with a power quality analyzer before rack power-on. Finally, a common commissioning error is failing to flush the coolant loop, leading to debris clogging the cold plates. Always flush and filter before connecting the rack.
Commissioning Sequence: Power, Coolant, and Network Bring-Up
Commissioning a GB300 NVL72 rack follows a strict sequence. At Leviathan Systems, we adhere to this order to prevent cascading failures. First, verify the facility power feed: measure phase-to-phase and phase-to-ground voltages at the rack PDU. Confirm the phase rotation matches the rack's label. Second, connect the coolant hoses and perform the pressure hold test. Third, power on the CDU and verify flow rate and temperature at the rack inlet. Fourth, power on the rack's PSUs one at a time, monitoring the power shelf for faults. Fifth, after the rack's management controller boots, check the coolant flow sensors and GPU temperatures via the BMC.
Sixth, install the scale-out network cables: clean, inspect, and test each MPO link before connecting to the switch and compute node. Seventh, power on the ToR switch and verify link lights on all ports. Eighth, run the OEM's system health check script to confirm all GPUs, NVLink, and network interfaces are online. Do not skip any step; a failure in one domain can cascade. For example, a coolant leak can cause a GPU to overheat and throttle, which will appear as a network performance issue. Document each step with time stamps and measurements.
Cable Management and Labeling: The Physical Layer's Documentation
In a GB300 NVL72 rack, the scale-out cable plant is dense. Each compute node has two MPO ports for the network, and the ToR switch has 64 or more MPO ports. Without disciplined labeling, troubleshooting becomes impossible. Use a label maker that prints durable, wrap-around labels. Label both ends of every trunk cable with a unique ID that matches the rack, row, and port number. For example: 'R04-U10-P1' for rack 4, compute node in U10, port 1. Also label the switch port and the compute node port with the same ID.
Use color-coded boots or sleeves to distinguish InfiniBand from Ethernet cables if both are present. Route cables in bundles of no more than 12, secured with Velcro straps at 1-meter intervals. Do not use zip ties; they can crush the fiber. Leave service loops at both ends to allow for re-routing or replacement. Document the cable routing in a spreadsheet or DCIM tool, including the test results for each link. This documentation is essential for future moves, adds, and changes.
Standards referenced: TIA-568.3-D (Optical Fiber Cabling and Component Standard) · IEC 61300-3-35 (Fibre Optic Connector Endface Inspection) · NEC Article 250 (Grounding and Bonding)
Frequently asked_
Can I use a standard 208V power feed for a GB300 NVL72 rack?
No. The GB300 NVL72 rack's power supply units are designed for 480V AC 3-phase. Using 208V will not provide enough power to run the rack at full load, and the PSUs may not operate correctly. The facility must provide a dedicated 480V circuit with a breaker sized per the OEM spec, typically 60A or 100A per rack.
What is the most common cause of MPO link failure in the field?
Contamination of the connector endface. Even a microscopic dust particle can cause high insertion loss or back-reflection. Always clean and inspect every MPO connector with a fiber microscope before mating, regardless of whether it came from the factory with a dust cap. Use a dry-click cleaner followed by a wet-dry wipe with isopropyl alcohol.
How do I test for a coolant leak before powering on the rack?
Leviathan Systems recommends performing a pressure hold test before power-on. After connecting the coolant hoses to the rack's blind-mate couplings, pressurize the loop to 1.5 times the maximum operating pressure specified by the OEM (typically 4.5 bar). Hold the pressure for 30 minutes. If the pressure drops more than 5%, there is a leak. Isolate the rack from the CDU and inspect the couplings and hoses.
Can I field-terminate MPO connectors on the scale-out cables?
No. MPO trunk cables are factory-terminated and polished. Field termination of MPO connectors is not recommended because it requires specialized equipment and cleanroom conditions to achieve the low insertion loss and endface quality required by TIA-568.3-D. All field work is limited to patching, routing, cleaning, inspection, and testing of pre-terminated assemblies.
What is the correct sequence for commissioning a GB300 NVL72 rack?
The sequence is: 1) Verify facility power feed and phase rotation. 2) Connect coolant hoses and perform pressure hold test. 3) Power on CDU and confirm flow rate and temperature. 4) Power on rack PSUs one at a time. 5) Check coolant sensors and GPU temperatures via BMC. 6) Install, clean, inspect, and test scale-out MPO cables. 7) Power on ToR switch and verify link lights. 8) Run OEM system health check. Do not skip steps; a failure in one domain can cascade into others.