Platforms_
GH200 Grace Hopper Deployment Guide: Cabling and Cooling a Superchip Node
Field procedures for integrating GH200 Grace Hopper nodes into racks, covering power feeds, internal NVLink connections, and liquid cooling loops with explicit differences from HGX baseboard layouts that change rack power sequencing and leak testing order.
Key facts
- GH200 places one Grace CPU and one Hopper GPU on a single module with NVLink-C2C between them rather than discrete GPUs on an HGX baseboard.
- Power delivery to a GH200 node requires separate CPU and GPU rails that must be brought up in a specific order before the rest of the rack.
- Liquid cooling on GH200 uses cold plates on both the CPU and GPU dies within the same module, creating a single thermal loop per node instead of per-GPU loops common on HGX.
- Scale-out fabric on GH200 racks uses MPO-terminated fiber between switches and nodes; NVLink remains copper and internal to each superchip.
- Factory-terminated MPO trunks require cleaning and inspection with a calibrated MPO continuity tester and OTDR before any node power-on sequence.
- Rack-level leak testing must occur after all node-level cooling connections are made but before any power is applied to the GPUs.
- OEM termination kits and torque specifications for power connectors must be followed exactly because mixed CPU/GPU rail layouts on GH200 create different inrush profiles than HGX.
Power rail sequencing for GH200 versus HGX
GH200 nodes split power between the Grace CPU rail and the Hopper GPU rail on the same module. Bring the CPU rail to nominal voltage first, confirm stable telemetry through the node BMC, then enable the GPU rail. This order prevents the GPU from attempting to train its memory controllers while the CPU side is still in reset.
HGX baseboards typically present a single high-current GPU domain per board. The GH200 split requires an extra set of PDUs or bus bars per node and changes the order in which breakers are closed during rack bring-up. Field crews label the two rail sets distinctly and verify with a calibrated multimeter at the node inlet before closing the second breaker.
NVLink topology inside the superchip
All NVLink lanes on a GH200 stay inside the module between the Grace CPU and Hopper GPU via the NVLink-C2C interface. No NVLink cables exit the node for GPU-to-GPU traffic. This removes the backplane NVLink spine work required on HGX eight-GPU trays.
Scale-out traffic moves exclusively over the InfiniBand or Ethernet fabric using MPO fiber between switches and nodes. Confirm that nvidia-smi reports the internal NVLink-C2C link as up before any fabric cables are connected; an external fiber problem cannot affect the on-module link.
Liquid cooling loop installation per node
Each GH200 carries two cold plates on one module. Route the supply and return lines so the CPU plate sees inlet fluid first, then the GPU plate. This order keeps the higher-power GPU on the colder side of the loop.
Manifold connections use the same quick-disconnect fittings as HGX racks, but the shorter hose runs per node reduce total fluid volume. Perform a low-pressure static test on each node loop before the rack-level fill to isolate any module-level leak from manifold joints.
Scale-out MPO patching and verification
MPO trunks for the compute fabric are factory-terminated. Field work consists of routing, cleaning the ferrules, inspecting with a scope, and testing continuity with a calibrated MPO tester followed by an OTDR trace. Any loss above the OEM limit requires the trunk to be replaced rather than re-terminated on site.
Patch panels at the top of rack must be labeled with both the switch port and the target node. Because GH200 nodes do not participate in an intra-rack NVLink fabric, the fiber plant can be fully tested and documented before any superchip receives power.
Rack integration sequence
Install nodes, connect power rails in CPU-then-GPU order, attach cooling loops, then terminate fiber. Only after all three domains pass inspection is the rack placed on the facility chilled-water circuit and leak-tested at operating pressure.
Commissioning begins with CPU-only power, followed by GPU rail enablement and internal NVLink-C2C verification. Fabric links are brought up last. This sequence prevents a cooling or fiber fault from masking a power-rail problem during initial telemetry collection.
Common failure modes observed in the field
The most frequent issue is reversed supply and return lines on a GH200 module, which places the GPU plate on the warmer side of the loop and triggers thermal throttling within minutes of load. Always verify flow direction with a temporary flow meter before final manifold connection.
Mixed rail labeling between CPU and GPU feeds has caused inrush trips when breakers are closed out of sequence. Another recurring fault is debris on MPO ferrules after trunk installation; an OTDR trace taken before node power-on catches these before they appear as CRC errors on the fabric. Finally, quick-disconnect fittings that are not fully seated pass a static test but weep under flow; a second pressure test after the pumps are running is required.
Standards referenced: TIA-568.3-D for fiber optic cabling and testing · OEM node installation manual for power rail sequencing and torque values · ASHRAE TC 9.9 liquid cooling guidelines for data center manifolds
Frequently asked_
Does GH200 require the same NVLink backplane cables as an HGX tray?
No. All NVLink on a GH200 stays inside the superchip module between the Grace CPU and Hopper GPU. No external NVLink cables or backplane spine are used. Scale-out traffic travels only over the MPO fiber fabric.
What is the correct order for applying power to a GH200 node?
Enable the Grace CPU rail first and confirm stable telemetry. Only then close the GPU rail. Reversing this order can prevent the GPU from completing memory training. Use labeled breakers and verify voltages at the node before proceeding.
How should the liquid cooling loop be routed on a GH200 module?
Connect so the CPU cold plate receives inlet fluid first, followed by the GPU plate. This keeps the higher-power die on the colder segment of the loop. Perform a node-level static pressure test before connecting to the rack manifold.
When should MPO fiber testing occur relative to node power-up?
Complete cleaning, inspection, and OTDR testing of all MPO trunks before any power is applied to the nodes. This isolates fabric problems from power or cooling issues during initial commissioning.
Who typically performs GH200 rack integration for large deployments?
Specialized field crews such as Leviathan Systems handle the mechanical, power, cooling, and fiber work because the split-rail power and combined cold plates change the standard HGX sequence.