Commissioning_
Thermal Soak & Burn-In During Commissioning: What to Watch
Details the sequence, monitoring points, and acceptance criteria for sustained-load thermal soak on H100-GB300 NVL72 racks to expose marginal cold plates, power supplies, and optics before sign-off.
Key facts
- GPU-to-GPU NVLink traffic stays on the internal copper spine; fiber and MPO paths carry only the scale-out IB or Ethernet fabric.
- Cold-plate loops must reach and hold the OEM-specified flow rate and differential pressure before any power is applied to the GPUs.
- Sustained soak duration is set by the rack OEM thermal validation procedure, measured from the point all cold-plate inlet temperatures stabilize.
- Power supplies are monitored for output voltage droop and fan-speed step changes under constant high load.
- Optics are inspected with a calibrated fiber microscope and cleaned with the OEM-approved cassette before any high-speed links are brought up.
- Leak detection sensors on the manifold and drip trays must remain dry for the entire soak window.
- Leviathan Systems records inlet/outlet temperatures, flow, pressure, and PSU telemetry at one-minute intervals throughout the test.
Rack Preparation Before Applying Sustained Load
Confirm that every cold-plate loop has been pressure-tested, flushed, and filled with the correct coolant mix. Verify manifold isolation valves are fully open and that secondary loop pumps are running at the design setpoint.
Install calibrated temperature probes on cold-plate inlets and outlets plus GPU cold-plate surface thermocouples where the OEM provides access ports. Connect the rack power distribution units to the facility feed and confirm all branch circuit breakers are rated for continuous load.
Route and secure the scale-out fiber trunks so they do not contact any coolant lines or sharp edges; label both ends before mating. MPO trunks arrive factory-terminated and must not be field-crimped.
Load Application Sequence
Begin with a low-power CPU-only workload to bring the system to a stable thermal state. Once inlet coolant temperatures have plateaued, step the GPUs to the first sustained power level specified in the OEM burn-in script.
Increase load in increments while watching differential pressure across each cold plate; any drop below the minimum indicates a flow restriction that must be corrected before continuing.
Hold the final power level for the full duration required by the rack validation procedure, recording all telemetry at one-minute intervals.
Monitoring Cold Plates and Liquid Cooling
Track inlet-to-outlet delta-T on every loop; an unexpected rise signals either reduced flow or a partial blockage inside the cold plate. Compare measured values against the OEM thermal model at the same power and flow.
Watch for any oscillation in pump speed or valve position that the control system did not command; such behavior often precedes a leak or air ingestion event.
At the end of the soak, perform a final visual inspection of every quick-disconnect and manifold joint while the system is still at operating temperature and pressure.
Power Supply and Electrical Stability Checks
Record input current balance across all three phases and confirm no single phase exceeds the continuous rating of its breaker. Log PSU output voltage and current for each rail; deviations greater than the OEM tolerance require replacement before handover.
Observe PSU fan curves; a fan that remains at maximum speed after temperature has stabilized usually indicates an internal hot spot or failing sensor.
Capture any PSU faults or warnings through the rack BMC; transient faults that clear on their own still count as failures for acceptance purposes.
Optics and Scale-Out Fabric Verification
After the thermal soak, re-inspect every MPO connector with a calibrated microscope. Look for scratches, pits, or contamination introduced by thermal expansion and contraction of the rack.
Run the full suite of scale-out link BER tests at the target data rate; any lane that required FEC margin reduction during the soak must be investigated and repaired.
Document the final cleaned and inspected state of each trunk so the next team knows the baseline condition at handover.
Common Field Failure Modes and How to Catch Them
Cold plates with marginal internal fins often pass initial flow tests but develop elevated delta-T only after several hours at full power; the only reliable detection method is continuous delta-T logging throughout the soak window.
Power supplies that were marginal at the factory sometimes pass short burn-in but trip or fold back when the rack reaches steady-state temperature; monitoring output voltage under constant load for the full duration catches these units.
Fiber end-faces can shift or collect particles when trunks are routed under thermal cycling; performing a post-soak inspection and clean before final BER testing prevents latent link flaps after the rack is in production.
Standards referenced: ASHRAE TC 9.9 thermal guidelines for data processing environments · OEM rack-level liquid cooling validation procedure · IEC 60529 IP ratings for enclosure integrity
Frequently asked_
How long should the soak run once full GPU power is reached?
Follow the exact duration stated in the rack OEM thermal validation document. The clock starts only after all cold-plate inlet temperatures have stopped changing by more than the stated tolerance. Shorter runs miss the slow drift failures that appear after several hours.
Do we need to exercise the NVLink fabric during the thermal soak?
NVLink traffic runs over the internal copper spine and does not depend on the fiber plant. The soak is intended to stress thermals and power; NVLink link-up checks are performed as part of the separate fabric commissioning sequence after the rack has passed thermal acceptance.
What constitutes a failed cold plate during soak?
Any loop whose delta-T exceeds the OEM model by more than the allowed margin at the measured flow and power, or any loop that shows a sudden pressure drop. The plate is replaced and the soak is restarted for that rack.
Who signs off on the soak results before Leviathan Systems hands the rack over?
The customer’s commissioning engineer reviews the logged telemetry against the acceptance criteria in the OEM procedure. Once both parties agree the data set is complete and within limits, the rack is released for production cabling and final integration.
Can we run the soak with only facility chilled water instead of the full CDU loop?
Only if the OEM procedure explicitly allows it and the facility supply can maintain the required inlet temperature and flow. Most NVL72-class racks require the production CDU to replicate the correct system impedance and control response.