Platforms_
HGX vs DGX: What's Different When You Deploy Them
A field-level comparison of HGX and DGX deployments, detailing the practical differences in rack assembly, GPU and bridge installation, liquid cooling integration, cabling, and commissioning for AI data centers.
Key facts
- DGX ships as fully integrated 8-GPU nodes with pre-installed NVLink bridges and pre-configured BMC; HGX baseboards require field installation of GPUs, bridges, and cooling hardware.
- HGX uses a passive midplane for NVLink interconnect; GPU modules must be seated with guide pins and torque-controlled fasteners per OEM spec. DGX uses an active NVSwitch on the motherboard.
- Liquid cooling for HGX involves field-attached cold plates with quick-disconnect (QD) fittings; DGX liquid-cooled variants have factory-sealed loops requiring only rack manifold connections.
- HGX racks have higher fiber density due to disaggregated switch-to-GPU connections (2–3× more MPO trunks per rack); DGX uses fewer, larger cables (OSFP direct-attach copper or active optical cables).
- Commissioning HGX requires per-GPU firmware updates and NVLink topology validation via nvidia-smi; DGX nodes have pre-validated firmware and a unified BMC for faster commissioning.
- Common failure modes in HGX include GPU seating errors, NVLink bridge misalignment, and coolant leaks at QD fittings; in DGX, firmware mismatches and network cable damage dominate.
- Relevant standards: TIA-568.3-D for fiber cabling polarity and insertion loss, ASME B31.3 for process piping (liquid cooling loops), and OEM installation guides for torque and TIM application.
Rack Assembly: HGX Requires Full GPU and Bridge Installation
HGX baseboards arrive as bare PCBs with GPU sockets, NVLink bridge slots, and power connectors. You must install each GPU module—typically eight per baseboard—using the OEM’s guide pins and a calibrated torque driver to apply even pressure across the socket. The torque specification (e.g., 0.6 N·m for certain sockets) must be followed exactly; over-torquing can crack the substrate, under-torquing leads to poor electrical contact. After the GPUs are seated, install the NVLink bridges onto the baseboard edge connectors. Each bridge is fragile; handle it by the edges and verify that its latch clicks fully closed. A partially seated bridge will produce NVLink link errors during validation.
DGX systems, by contrast, ship as complete nodes with GPUs, NVLink bridges, and the NVSwitch ASICs already integrated and factory-tested. Rack assembly consists of sliding the node into the rack rails, securing it with captive screws, and connecting power and network cables. There is no field-level GPU or bridge installation, which reduces assembly time per rack by roughly 40–60% compared to an equivalent HGX build. However, DGX nodes are heavier—often exceeding 35 kg—and require a mechanical lift or two-person lift for safe installation. The trade-off is speed and simplicity versus the ability to customize hardware at the component level.
Liquid Cooling Integration: Field-Attached vs. Factory-Sealed Loops
For HGX systems with direct-to-chip liquid cooling, you must attach cold plates to each GPU and, on some baseboards, to memory modules. This begins with applying thermal interface material (TIM) in the OEM-specified pattern—typically a cross or dot pattern—using a stencil for consistency. The cold plate is then secured with a torque sequence that ensures even compression. After the cold plate is mounted, connect it to the rack-level coolant loop via quick-disconnect (QD) fittings. Each QD fitting must be cleaned with isopropyl alcohol and inspected for O-ring damage before mating. Once all connections are made, pressure-test the loop at 1.5× the operating pressure (per the CDU manufacturer’s procedure) and hold for 30 minutes; a pressure drop of more than 5% indicates a leak.
DGX liquid-cooled variants, such as the DGX H100 liquid-cooled, have factory-sealed cooling loops that include the cold plates, tubing, and QDs. The only field connection is at the rack manifold—two hoses per node. This eliminates the risk of TIM application errors and reduces the number of potential leak points by an order of magnitude. However, you must still perform a leak test at the manifold connection using a pressure decay method or flow verification through the CDU’s monitoring interface. The trade-off: a failed cold plate in a DGX requires returning the entire node, whereas an HGX cold plate can be swapped in the field. For large deployments, Leviathan Systems recommends stocking spare cold plates and QD fittings for HGX, and spare nodes for DGX.
Cabling and Networking: Density and Polarity Differences
HGX racks typically have higher fiber density because each GPU connects to top-of-rack (ToR) switches via individual or breakout cables. For a four-GPU HGX baseboard, you might have four to eight MPO-12 or MPO-24 trunks per baseboard, depending on the network architecture (e.g., InfiniBand NDR400 or Ethernet 800G). These cables must be routed through cable management arms (CMAs) with bend-radius protection as specified in TIA-568.3-D. Polarity must be verified with a calibrated MPO continuity tester; the choice of polarity (A, B, or C) depends on the transceiver and switch specifications. MPO trunks are factory-terminated—never field-crimped—so any damage requires replacing the entire trunk.
DGX systems use fewer, thicker cables because the node’s internal NVSwitch handles GPU-to-GPU traffic, and only the node’s network interfaces (e.g., eight OSFP ports) connect to the ToR switches. This reduces the number of fiber trunks per rack by roughly 50%. However, DGX cables are often OSFP-to-OSFP direct-attach copper (DAC) or active optical cables (AOC), which are less flexible than MPO trunks and require larger bend radii—typically 30 mm for DAC and 15 mm for AOC. You must also ensure the cable length matches the rack-to-switch distance; excessive length adds weight, obstructs airflow, and increases signal attenuation. Label both ends of every cable with the source and destination ports to simplify troubleshooting.
Commissioning and Validation: Per-GPU vs. Per-Node Testing
Commissioning an HGX rack requires testing each GPU individually. After power-on, run nvidia-smi to confirm that all eight GPUs are detected, then use nvidia-fabricmanager to validate the NVLink topology. You must check that each NVLink bridge reports the correct link width—typically 400 GB/s per bridge for H100—and that no links are degraded (e.g., showing half width). Firmware updates are applied per GPU using nvidia-fwupd, which can take 30–60 minutes per rack. Finally, run a stress test such as NCCL all-reduce to verify bandwidth and latency across the rack; compare results to the expected values for your GPU count and NVLink configuration.
DGX nodes come with pre-validated firmware and a unified BMC that reports node health. Commissioning involves powering on the node, connecting to the BMC web interface, and running the built-in diagnostics. The diagnostics check GPU memory, NVLink, and network interfaces in a single pass, typically completing in 15–20 minutes per node. You still run a network-level stress test across the cluster, but the per-node validation is faster. The trade-off is that if a DGX node fails diagnostics, you must replace the entire node, whereas an HGX rack allows swapping a single GPU or bridge. For hyperscale deployments, Leviathan Systems has found that DGX reduces commissioning time per rack by 50–60% compared to HGX.
Common Failure Modes in the Field and How to Catch Them
In HGX deployments, the most common failure is a partially seated GPU module. This causes the GPU to not appear in nvidia-smi or to report correctable errors (e.g., ECC single-bit errors). Catch this by visually inspecting the socket latch alignment and using a continuity test on the power and ground pins before power-on. Another frequent issue is NVLink bridge misalignment—the bridge’s edge connector may not fully engage with the baseboard slot, leading to degraded link width. Use a borescope to inspect the bridge-to-slot interface if the link width is lower than expected; a misaligned bridge will show a gap of more than 0.5 mm.
For liquid-cooled HGX systems, QD fitting leaks are common if the O-ring is damaged during connection. Always inspect the O-ring with a magnifying glass before mating, and perform a pressure hold test after all connections are made. In DGX systems, the most common failure is a firmware mismatch between the node’s BMC and the GPU firmware, which can cause the node to fail diagnostics or become unresponsive. Always update the DGX node firmware to the latest version from the OEM’s portal before commissioning. Network cable damage—especially at the connector—is a risk in both systems; use an OTDR to measure insertion loss and detect breaks. A damaged OSFP connector can cause link errors that are intermittent, so test all cables before installation.
Decision Criteria: Choosing HGX or DGX for Your Deployment
Choose HGX if you need maximum flexibility in GPU selection, cooling design, or network topology. HGX allows you to mix GPU SKUs on the same baseboard (if supported by the OEM) and use your own CDU or network switches. It also enables field repair of individual components, which can reduce downtime if you have spare GPUs, bridges, or cold plates on hand. However, HGX requires a skilled field crew with experience in GPU seating, bridge installation, and liquid cooling loop assembly. The learning curve is steep, and first-time deployments often see higher rework rates.
Choose DGX if you prioritize deployment speed and reliability. DGX systems are pre-integrated and tested, reducing the risk of assembly errors. They are ideal for hyperscale deployments where time-to-production is critical, such as one of the largest hyperscalers in Texas, which uses DGX for rapid cluster rollouts. DGX also simplifies supply chain logistics because you order a complete node rather than separate baseboards, GPUs, and bridges. The trade-off is higher per-node cost and less flexibility in customization. For most AI data centers, the decision comes down to whether you have the in-house expertise for HGX assembly or prefer the turnkey approach of DGX.
Standards referenced: TIA-568.3-D (Optical Fiber Cabling and Components Standard) · ASME B31.3 (Process Piping) · IEC 61753-1 (Fiber Optic Connector Performance) · NVIDIA NVLink Bridge Installation Guide (per OEM spec)
Frequently asked_
Can I use the same liquid cooling loop for both HGX and DGX racks?
Yes, the rack-level CDU and coolant distribution manifold can be the same, but the connection points differ. HGX requires field-attached QD fittings at each GPU cold plate, while DGX uses factory-sealed loops with QDs at the node level. You must ensure the CDU's flow rate and pressure match the OEM's specifications for each system. For mixed deployments, use separate manifolds or pressure-regulating valves to avoid over-pressurizing the DGX loops. Leviathan Systems recommends a minimum of 1.5× pressure margin for the HGX loop connections.
How do I verify NVLink topology in an HGX rack after assembly?
After power-on, run 'nvidia-smi topo -m' to display the NVLink connectivity matrix. Each GPU should show links to its paired GPUs (e.g., GPU0 to GPU1). Then use 'nvidia-fabricmanager -validate' to check link width and error counts. If any link shows degraded width (e.g., 200 GB/s instead of 400 GB/s), inspect the NVLink bridge seating and reseat if necessary. For a full validation, run an NCCL all-reduce benchmark and compare bandwidth to the expected value for your GPU count.
What is the typical time savings of deploying DGX over HGX per rack?
Based on field experience, a DGX rack can be assembled and commissioned in roughly 40–50% less time than an equivalent HGX rack. For example, a four-node DGX H100 rack might take 4–6 hours for physical installation and 2–3 hours for commissioning, while an HGX rack with four baseboards (32 GPUs) could take 8–12 hours for assembly and 4–6 hours for commissioning. The exact savings depend on crew experience, the complexity of liquid cooling connections, and whether the HGX baseboards require firmware updates.
Can I field-replace a GPU in a DGX node?
No, DGX nodes are not designed for field GPU replacement. The GPUs are soldered or socketed with a proprietary retention mechanism that requires factory-level rework. If a GPU fails, you must replace the entire DGX node. This is a key consideration for maintenance planning; you should stock spare DGX nodes rather than spare GPUs. For HGX, you can replace individual GPUs in the field using the OEM's removal tool and thermal interface material kit. Leviathan Systems always keeps a spare GPU kit per 100 racks when deploying HGX.