LEVIATHAN SYSTEMS

Commissioning_

Thermal Soak & Burn-In During Commissioning: What to Watch

·Technical content·

Interpret sustained-load thermal observations, compare power and cooling conditions, and separate symptoms from diagnosis before a commissioning handoff.

Key facts

  • Name the actual system: NVIDIA DGX H100/H200 eight-GPU systems and DGX GB NVL72 rack architectures are different references.
  • Comparable observations need the same equipment and configuration, identified sensor points, workload, observed power, cooling conditions and time basis.
  • A coolant temperature-rise change depends on heat transfer, flow, fluid properties and operating state; it does not by itself identify a failed cold plate.
  • A pressure observation needs its circuit, reference, measurement points, flow and control conditions before it can support a hydraulic comparison.
  • NVLink coverage follows the selected validation checks; copper transport does not justify excluding it from the required coverage.
  • Optical link events need endpoint, module, metric and observation context. Finishing a soak does not create a universal reason to disconnect and clean every MPO connection.
  • An observation, a possible explanation, a component diagnosis and an authorized disposition are separate conclusions.

Identify the Equipment and the Observation Being Reviewed

A sustained-load thermal record can show how identified equipment behaved during a stated interval under recorded conditions. Its value at handoff depends on whether those observations answer the actual requirement. A changing trace can justify investigation without identifying a defective part; a steady trace can support its stated observation without guaranteeing future reliability.

Begin with the supplied system rather than a range of GPU generation names. NVIDIA describes DGX H100/H200 as eight-GPU systems. Its DGX GB rack guide separately describes NVL72 racks with 72-GPU NVLink domains. In that named DGX GB architecture, CPU and GPU cold plates coexist with air-cooled networking and storage components in the compute trays. Those descriptions do not establish the arrangement of every OEM system using the same GPU family. [N1, N2]

For a comparison, retain the equipment identity and configuration, workload, observed power, coolant or air measurement locations, relevant flow and control state, and observation interval. Compare measured values against an applicable OEM reference at the corresponding conditions. A rack-return temperature and a cold-plate outlet temperature describe different boundaries even when both are reported in degrees Celsius.

This article explains interpretation of returned observations. The applicable OEM and site method governs actual operation, responses and acceptance criteria. The existing Thermal Burn-In for GPU Clusters article develops the broader duration and test-coverage questions; the focus here is what a particular observation establishes before handoff.

Keep Load, Elapsed Time and Interconnect Coverage Together

A full-load label is insufficient to describe the work that produced a trace. NVIDIA's DCGM Targeted Stress reference describes a configured compute-performance objective, while Targeted Power describes a configured board-power objective under sustained compute load. Similar elapsed times therefore need not represent the same objective or coverage. Neither plugin supplies a universal rack-soak programme. [D2, D3]

Identify which checks actually completed on which resources. NVIDIA's Diagnostics reference includes NVLink, compute, power and thermal paths among its possible coverage, with availability dependent on the release, plugins, hardware and environment. The passive copper NVLink backplane in the named DGX GB design describes transport, not permission to omit NVLink from the required checks or defer it automatically until after thermal acceptance. [D1, N2]

Duration also needs its actual start condition, interruptions and workload transitions. Follow the equipment-specific requirement rather than assuming a CPU-first sequence or starting every soak clock at a generic coolant plateau. Keep the observation window distinct from the collection interval: sparse samples cannot establish every event between them. The record should state its actual sampling and event coverage; this article prescribes neither one-minute logging nor additional probe installation.

Interpret Coolant Temperature and Pressure at Their Measured Boundaries

Define the temperature difference before interpreting its trend. Return minus supply within one coolant circuit describes its temperature rise across the named boundary. OCP's Liquid to Liquid CDU Test Methodology and Performance Rating instead defines approach temperature difference as technology-cooling-system supply minus facility-water-system supply. These are different comparisons. The OCP document concerns CDU qualification and performance reporting. [O1]

For a simplified, near-steady single-phase heat balance, the heat carried by a coolant stream depends on mass flow, specific heat and temperature rise. More heat transferred into the same flow can increase the rise; less flow carrying the same heat can also increase it. Fluid properties and transient heat storage matter when comparing actual observations. Eaton's heat-load explanation supports this relationship, not a rack acceptance threshold. GPU electrical power should not be silently substituted for heat entering a particular liquid boundary. [E1]

Likewise, distinguish a pressure drop across two named points from a fall in pressure at one sensor. Retain the circuit, pressure reference, ordered measurement points, corresponding flow and pump or valve control conditions. OCP treats technology-side pressure head and facility-side impedance separately. A changed differential at changed flow does not isolate a new restriction, and a pressure change alone does not identify a failed cold plate. [O1]

Eaton's loop-design discussion distinguishes deposited material restricting flow from fouling adding thermal resistance at heat-transfer surfaces. These are candidate mechanisms requiring applicable diagnosis, not interchangeable meanings of a rising temperature. Pump or valve oscillation and leak indications also need their own event context; oscillation alone does not establish an impending leak. Keep any actual leak alarm or observed liquid visible as a separate finding for the responsible party. [E2]

Read Power and Fault Events Without Selecting a Repair

Retain the reported power boundary, field definition, units and time basis alongside the workload. A GPU board reading, a PSU output reading and a rack input reading answer different questions. A change in one should not be relabelled as a measured change in another.

Targeted Power provides a useful interpretation example: its reference distinguishes failure to reach the configured board-power threshold from an enforced power limit that makes the target unattainable. The latter is a configuration constraint. Neither outcome by itself proves a defective PSU. Keep the actual result and reported reason rather than reducing both to insufficient power. [D3]

Fan-speed changes and platform faults remain useful observations. A fan remaining at maximum speed does not, by itself, identify a failed sensor. A fault that clears still belongs in the event history, with its timestamp, reported severity, associated conditions and recovery information. Whether it affects acceptance follows the applicable event guidance and agreed criteria; neither automatic rejection nor dismissal follows simply from its clearing. DCGM itself does not repair faults or determine RMA eligibility. [D1]

Keep Optical Link Evidence Separate from an Assumed Thermal Cause

When a link event occurs during the observation window, identify the actual endpoints and transceiver modules, configured link rate, reported event or metric, units and interval. Compare the same field under known conditions. A statement such as reduced FEC margin is insufficient without the actual metric definition, comparison basis and applicable criterion; it does not specify a repair.

A link event coinciding with a temperature change establishes a timing relationship, not proof that rack thermal expansion damaged an end-face. Connector condition, a link's operational reports and any separately scoped optical measurement are different evidence. Preserve each with its actual method and coverage rather than using a thermal trace to claim that the fiber plant passed or failed.

Fluke Networks' November 21, 2019 troubleshooting article warns that disconnecting and cleaning an already-clean connection can introduce contamination. The end of a soak therefore supplies no universal mandate to remate or clean every MPO connector. Keep any actual inspection findings and approved corrective work associated with the affected connection. Further inspection or instrument testing belongs to its defined task, resources and criteria. [F1]

Carry the Observation Through Diagnosis to the Handoff Decision

Consider two fictional returns for the same rack. Return A reports a whole-rack coolant return-minus-supply difference at one workload. Return B reports a branch-level difference with higher recorded GPU power at another workload. B's larger temperature difference is described in the summary as a worsening cold plate. No real readings, test outcomes or operating targets are represented.

That conclusion skips two comparisons: the measurement boundary changed, and so did the workload and observed power. The larger difference is the observation. Changed heat transfer or flow is a possible explanation. Identifying a defective cold plate requires an applicable technical diagnosis. Replacing it, repeating work or accepting the evidence is a separate disposition by the appointed owner. The submitted comparison establishes none of those later conclusions. Restoring comparable conditions and the applicable OEM comparison is the missing analytical step; the example does not prescribe a rerun.

At handoff, retain useful results with those distinctions intact. Keep the original observations, the specialist's explanation and supporting evidence, the affected requirement, and the actual accepting disposition or unresolved question. If authorized follow-up changes equipment or conditions, connect its return to the earlier finding rather than treating a later favorable graph as an explanation of the original event. The project's arrangements identify the reviewers and acceptance authority; this article assigns no witness or sign-off role.

Leviathan's Hardware Burn-In & Validation service can address an agreed hardware-QA question with the configuration, method, resources, duration, acceptance parameters and results record confirmed for the engagement. Physical cabling follow-up can use the Physical-Layer Verification Services link in Related; formal network tests require separately confirmed instruments, personnel, methods and criteria. Liquid-side work, facility electrical work and full system/software commissioning are outside this physical service scope.

Use Discuss a Project to request Hardware Burn-In & Validation or defined physical verification. Share a nonconfidential equipment and configuration summary, the observation to be resolved, the decision the evidence must support and the relevant site constraints. That gives the scope discussion a concrete question without promising a diagnosis, a particular test result or whole-system release.

Standards referenced: N1 — NVIDIA DGX H100/H200 User Guide, Introduction to NVIDIA DGX H100/H200 Systems — https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html — current unversioned page checked 2026-10-20; eight-GPU system identity, not an NVL72 rack procedure. · N2 — NVIDIA DGX GB Rack Scale Systems User Guide, Hardware — https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html — current unversioned page checked 2026-10-20; named DGX GB200/GB300 NVL72 architecture, passive copper NVLink backplane and compute-tray heat paths; other OEM implementations require their own references. · D1 — NVIDIA DCGM Diagnostics, Goals, Beyond Scope, Run Levels and Tests — https://docs.nvidia.com/datacenter/dcgm/latest/learn/modules/dcgm-diagnostics.html — latest documentation, page updated 2026-09-10, checked 2026-10-20; possible diagnostic coverage, environment-dependent availability and limits of diagnosis/RMA conclusions. The installed release is not assumed. · D2 — NVIDIA DCGM Targeted Stress Plugin — https://docs.nvidia.com/datacenter/dcgm/latest/reference/diagnostics/plugins/targeted-stress.html — latest documentation, page updated 2026-09-10, checked 2026-10-20; configured compute-performance objective, not a generic rack-soak duration or threshold. · D3 — NVIDIA DCGM Targeted Power Plugin — https://docs.nvidia.com/datacenter/dcgm/latest/reference/diagnostics/plugins/targeted-power.html — latest documentation checked 2026-10-20; board-power objective and distinction between target failure and an unattainable target caused by an enforced power limit. No default or command is adopted. · O1 — Open Compute Project, Liquid to Liquid CDU Test Methodology and Performance Rating, Revision 1.0, sections 3.3.2–3.3.2.4 — https://www.opencompute.org/documents/ocp-wp-l-lcdu-test-methodology-performance-rating-r1-pdf — checked 2026-10-20; page headers say August 2024 and the version table prints 11/01/2024. Supports circuit-specific thermal and pressure/flow reporting, not rack-soak rules or cold-plate replacement criteria. · E1 — Eaton, How to measure your heat load — https://www.eaton.com/gb/en-gb/products/thermal-management-solutions/measurement-limits/measuring-your-heat-load.html — undated page checked 2026-10-20; qualitative relationship among coolant heat transfer, mass flow, specific heat and temperature rise. No sensor installation method or numerical target is adopted. · E2 — Eaton, High performance liquid cooling loop design, Corrosion prevention — https://www.eaton.com/us/en-us/products/thermal-management-solutions/liquid-cooling/high-performance-liquid-cooling-loop-design.html — undated page checked 2026-10-20; distinguishes flow restriction from additional thermal resistance caused by fouling; does not diagnose an actual component. · F1 — Fluke Networks, Troubleshooting Fiber on the Cheap, published 2019-11-21 — https://www.flukenetworks.com/blog/cabling-chronicles/troubleshooting-fiber-cheap — checked 2026-10-20; unnecessary disconnecting and cleaning can contaminate a clean connection; does not prescribe post-soak treatment or an optic acceptance method.

Frequently asked_

How long should the soak run once full GPU power is reached?

Use the duration and start condition in the applicable equipment-specific validation requirement. The return needs its actual workload, operating conditions, interruptions and covered interval. Reaching a power level or observing a coolant plateau does not create a universal clock-start rule, and elapsed hours alone do not establish complete coverage.

Do we need to exercise the NVLink fabric during the thermal soak?

The selected validation plan determines the required coverage. NVIDIA documents diagnostics that can exercise NVLink alongside other paths, subject to release, hardware and environment. Copper transport in a named architecture does not justify excluding it. Verify what actually completed rather than inferring coverage from the word soak.

What constitutes a failed cold plate during soak?

A temperature rise or pressure change is an observation requiring interpretation, not a standalone cold-plate failure definition. Establish the actual measurement boundary, heat/flow and control conditions, applicable OEM criteria and supporting diagnosis. Neither automatic replacement nor restarting the entire soak follows from that symptom alone.

Who decides what the soak observations establish at handoff?

The actual project arrangements identify the technical reviewer and acceptance authority for each scope. The return should distinguish observations, diagnosis, unresolved exceptions and the accepting disposition. Delivering a report or witnessing an activity does not automatically release the whole system or assign that authority to Leviathan.

Can observations from a temporary cooling arrangement establish the production configuration?

Keep the arrangement and conditions that actually produced the observations. A temporary setup's result does not automatically establish behavior with a different CDU, circuit or control arrangement. The appointed OEM, cooling and facility parties determine applicability and any further evidence required; this article supplies no bypass or chilled-water connection procedure.

Discuss Your Infrastructure Project_

Tell us about your project, and our team will follow up to discuss the scope, requirements and availability.

Discuss a ProjectEmail Project Scope