Platforms_
NVIDIA GB300 NVL72 Explained: Specs, Power, and What It Takes to Deploy
A field engineer’s guide to the NVIDIA GB300 NVL72 rack: its architecture, power/cooling specs, and the physical-layer deployment steps (rack assembly, liquid cooling, copper NVLink backplane, MPO fiber patching) that determine whether the system works at rated performance.
Key facts
- GB300 NVL72 integrates 72 Grace CPUs and 72 Blackwell GPUs in a single rack, with NVLink 5 running over a copper backplane inside the rack; fiber/MPO carries only the scale-out network (InfiniBand or Ethernet).
- The rack’s power draw exceeds 120 kW at full load, requiring 48V DC bus bars and liquid cooling for both CPUs and GPUs; air cooling is insufficient for the GPU thermal design power.
- The copper NVLink backplane is a passive printed-circuit-board assembly with 1,728 differential pairs per rack; field work is limited to seating GPU trays and verifying link training via nvidia-smi — no fiber or MPO touches NVLink.
- Liquid cooling uses a rear-door heat exchanger (RDHx) or direct-to-chip cold plates with a facility-side coolant distribution unit (CDU); the rack’s quick-disconnect fittings must be leak-tested at the OEM-specified pressure before power-on.
- MPO trunk cables for the scale-out network are factory-terminated and polished; field patching requires cleaning with a one-click cleaner, inspecting with a 200x or 400x scope, and testing with a calibrated MPO continuity tester or OTDR.
- The rack’s structural weight exceeds 1,500 kg fully loaded; floor loading must be verified per the facility’s raised-floor load rating, and the rack must be anchored to seismic brackets in any region with seismic risk.
- Commissioning requires a phased power-up: first the CDU and coolant loop, then the rack’s PSUs and bus bars, then the GPU trays one at a time, monitoring for overcurrent and thermal events before enabling the full NVLink fabric.
GB300 NVL72 Architecture: What Lives Inside the Rack
The GB300 NVL72 is a single-rack system containing 72 Grace CPUs and 72 Blackwell GPUs, interconnected by NVLink 5 over a copper backplane that runs the full height of the rack. This backplane is a passive PCB assembly with 1,728 differential pairs — no active retimers, no fiber, no MPO. The copper traces carry GPU-to-GPU traffic at the NVLink 5 data rate, which is roughly double the per-lane rate of NVLink 4. The backplane is factory-assembled and tested; field work is limited to seating the GPU trays into their slots and verifying link training via nvidia-smi.
The scale-out network (InfiniBand or Ethernet) is a separate domain. Each GPU connects to a front-end network interface card (NIC) that plugs into a leaf switch at the top of the rack or in an adjacent row. These NIC-to-switch links use MPO fiber trunks — typically 12-fiber or 24-fiber MPO-12 or MPO-24 connectors — carrying 800 Gbps or 1.6 Tbps per port. The fiber trunks are factory-terminated; field work is patching, routing, cleaning, inspection, and testing. Never conflate the copper NVLink backplane with the fiber scale-out network — they are electrically and logically independent.
Power Delivery: 48V Bus Bars and the 120 kW+ Draw
The GB300 NVL72 rack draws over 120 kW at full load, which is too high for traditional 208V or 240V AC distribution. The rack uses a 48V DC bus bar system fed by multiple power supplies (PSUs) that convert facility AC (typically 480V 3-phase) to 48V DC. The bus bars run vertically inside the rack, and each GPU tray taps into them via hot-swappable connectors. The PSUs are redundant (N+1 or 2N) and must be wired per the OEM’s torque spec for the bus-bar lugs — overtightening can crack the bus bar, undertightening causes arcing.
The facility must provide a dedicated 480V 3-phase circuit with a breaker rated for the rack’s full load plus a safety margin per the National Electrical Code (NEC) or local equivalent. The rack’s power distribution unit (PDU) includes a branch circuit monitor that logs current per phase; during commissioning, verify that the phase imbalance stays within the OEM’s spec (typically under 10%). If the facility uses a UPS, ensure the UPS can handle the inrush current when the rack’s PSUs power on — inrush can exceed 200 A per phase for a few milliseconds.
Liquid Cooling: RDHx, Cold Plates, and Leak Testing
The GB300 NVL72 requires liquid cooling for both the Grace CPUs and the Blackwell GPUs. The rack ships with a rear-door heat exchanger (RDHx) that connects to the facility’s chilled water loop via a coolant distribution unit (CDU). The CDU pumps a dielectric coolant (typically a propylene-glycol/water mix) through cold plates mounted on each GPU and CPU die. The cold plates are pre-installed on the trays; field work is connecting the flexible hoses from the tray to the rack’s manifold using quick-disconnect fittings.
Before power-on, the entire coolant loop must be leak-tested at the OEM-specified pressure (typically 1.5x the operating pressure) for a minimum of 30 minutes. Use a calibrated pressure gauge and a leak-detection spray (e.g., a soap solution) on every fitting. After the pressure test, flush the loop with deionized water to remove any debris from the hoses or fittings. The CDU’s flow rate and inlet temperature must be set per the OEM’s spec — typically a flow rate of 10–15 liters per minute per tray and an inlet temperature of 20–25 °C. If the facility’s chilled water is above 25 °C, the CDU will need a supplemental chiller or the rack will throttle.
Structured Cabling: MPO Fiber Routing and Bend Radius
The scale-out network uses MPO trunk cables — typically 12-fiber or 24-fiber OM4 or OM5 multimode, or single-mode for longer reaches. These trunks are factory-terminated and polished; field work is routing them from the NICs at the front of the rack to the leaf switches in the adjacent row or overhead cable tray. The minimum bend radius for a loaded MPO trunk is specified by the cable manufacturer — typically 10x the cable diameter under load and 15x when not under load. Exceeding the bend radius causes micro-bends that increase attenuation and can cause link errors or complete loss of signal.
Every MPO connector must be cleaned with a one-click cleaner (a dry, lint-free tool that wipes the ferrule end-face) and inspected with a 200x or 400x fiber-optic microscope before mating. Use a calibrated MPO continuity tester to verify polarity (method A, B, or C per TIA-568.3) and a power meter and light source to measure end-to-end loss. The loss budget for a single-mode MPO link at 1.6 Tbps is typically under 1.0 dB per connector pair; for multimode, under 0.75 dB. If a link fails, re-clean and re-inspect — 90% of failures are due to contamination or damage on the ferrule.
Common Failure Modes: What Goes Wrong in the Field
The most common failure in GB300 NVL72 deployment is coolant leaks at the quick-disconnect fittings. The fittings are designed for thousands of mate/unmate cycles, but debris from the hoses or a misaligned tray can damage the O-ring, causing a slow drip. Catch it by running the pressure test for the full 30 minutes and checking every fitting with a leak-detection spray. A second common failure is MPO fiber contamination: a single speck of dust on a 12-fiber MPO ferrule can block one or more fibers, causing link errors. Always clean and inspect before mating, and never touch the ferrule end-face.
A third failure mode is bus-bar lug overtightening. The torque spec for the bus-bar lugs is typically 15–20 Nm (check the OEM spec); using an impact driver without a torque limiter can crack the bus bar, leading to arcing and a fire risk. Use a calibrated torque wrench and tighten in a star pattern. A fourth failure is GPU tray seating: the tray’s edge connector must fully engage with the backplane. If the tray is not fully seated, the NVLink links will not train. Verify by checking the tray’s latch mechanism clicks into place and by running nvidia-smi to confirm all 72 GPUs are visible. Finally, the rack’s weight can exceed the floor’s load rating — always verify the raised-floor load capacity before rolling the rack into position. If the floor is rated for 1,200 kg per tile and the rack weighs 1,500 kg, you need a load-spreading plate or a reinforced tile.
Commissioning Sequence: Phased Power-Up and Validation
Commissioning a GB300 NVL72 rack follows a strict sequence to avoid damaging components. Step 1: Verify the facility’s chilled water loop is flowing at the correct temperature and pressure, then connect the CDU and leak-test the coolant loop. Step 2: Power on the CDU and circulate coolant for at least 15 minutes to stabilize temperatures. Step 3: Connect the facility’s 480V 3-phase power to the rack’s PDU and verify the bus-bar voltage is 48V DC ± 1%. Step 4: Power on the rack’s PSUs one at a time, monitoring the branch circuit monitor for overcurrent. Step 5: Insert the first GPU tray, power it on, and check nvidia-smi for GPU visibility and NVLink link training. Repeat for each tray, verifying that the NVLink fabric shows all 72 GPUs as connected.
After all trays are powered, run a fabric stress test (e.g., NCCL all-reduce) to validate the NVLink bandwidth and the scale-out network. If the scale-out network uses InfiniBand, run ibdiagnet to check for link errors. If any link shows errors, inspect the MPO connector and re-clean. Finally, monitor the coolant temperature and flow rate for 30 minutes under load to ensure the CDU can maintain the inlet temperature within spec. If the temperature rises above the OEM’s threshold, the GPUs will throttle — check the facility’s chilled water supply temperature and the CDU’s pump speed.
Standards referenced: TIA-568.3 (MPO polarity and loss testing) · National Electrical Code (NEC) Article 645 (data center power) · OEM-specific torque spec for bus-bar lugs · OEM-specific coolant pressure test spec (1.5x operating pressure)
Frequently asked_
Does the GB300 NVL72 use fiber for NVLink?
No. NVLink 5 in the GB300 NVL72 runs over a copper backplane inside the rack. The backplane is a passive PCB assembly with 1,728 differential pairs. Fiber/MPO cables carry only the scale-out network (InfiniBand or Ethernet) between racks and switches. Never confuse the two — nvidia-smi 'NVLink up' depends on the copper backplane, not on any fiber link.
What is the minimum floor load rating for a GB300 NVL72 rack?
The rack weighs over 1,500 kg fully loaded. The facility’s raised-floor load rating must be at least that per tile, plus a safety margin. If the floor is rated for 1,200 kg per tile, you need a load-spreading plate or a reinforced tile. Always verify the floor’s load capacity before rolling the rack into position.
How long does it take to commission a GB300 NVL72 rack?
A typical deployment by an experienced crew like Leviathan Systems takes 2–3 days: one day for rack assembly and liquid cooling hookup, one day for structured cabling and MPO patching, and one day for phased power-up and validation. The actual time depends on the facility’s readiness (chilled water, power, cable trays) and the number of racks being deployed simultaneously.
What tools are required for field deployment of the GB300 NVL72?
You need a calibrated torque wrench for bus-bar lugs, a pressure gauge and leak-detection spray for coolant loop testing, a one-click cleaner and 200x/400x fiber-optic microscope for MPO connectors, a calibrated MPO continuity tester and power meter/light source for fiber loss testing, and a digital multimeter for verifying bus-bar voltage. Also have a torque screwdriver for rack-mounting screws and a cable tension meter for MPO trunk routing.
Can the GB300 NVL72 be air-cooled?
No. The thermal design power of the Blackwell GPUs and Grace CPUs exceeds what air cooling can handle in a dense rack. The rack requires liquid cooling via a rear-door heat exchanger or direct-to-chip cold plates. Attempting to air-cool the rack will cause immediate thermal throttling and potential damage to the components.