Testing_
Thermal Burn-In for GPU Clusters: Duration, Watch Items, Pass/Fail
A field-proven protocol for thermal burn-in of GPU clusters (H100 and newer architectures in NVL72 racks), specifying soak duration, critical watch items, and objective pass/fail criteria to detect marginal hardware and cooling faults before production deployment.
Key facts
- Thermal burn-in for GPU clusters should run a minimum of 4 hours at sustained 100% GPU utilization (compute + memory) to stabilize junction temperatures and reveal thermal creep.
- GPU junction temperature (Tj) must never exceed the OEM-specified maximum (check your GPU's spec sheet, typically 85–95°C) at any point during the soak; any exceedance is an automatic fail.
- Delta between GPU inlet air temperature and exhaust air temperature across a rack should not exceed 15°C under full load per common rack manufacturer guidelines; larger deltas indicate airflow bypass or blocked vents.
- NVSwitch temperature on NVL72 racks must stay below its OEM threshold (typically 85°C); a rise of more than 5°C above ambient after 2 hours of soak indicates inadequate liquid cooling flow (verify with your specific switch spec).
- Power draw per GPU should stabilize within ±5% of the TDP spec after the first 30 minutes; a drifting downward trend signals thermal throttling or a failing VRM.
- Liquid cooling loop delta-T (supply vs. return) must remain within the CDU manufacturer's spec (typically 10–15°C at full load); a sudden increase of >3°C above baseline indicates a blocked cold plate or pump degradation.
- All GPU memory errors (correctable and uncorrectable) must be zero during the final 2 hours of soak; any non-zero count after the first 30 minutes warrants investigation and is a fail.
Burn-In Duration: Why 4 Hours Is the Minimum
Thermal burn-in for GPU clusters—whether air-cooled H100s or liquid-cooled NVL72 racks—requires a sustained full-load soak of at least 4 hours. Shorter runs miss thermal creep: the slow rise in junction temperature as thermal paste cures, cold plates settle, or airflow paths change under heat. In our field experience at Leviathan Systems, the first 90 minutes often look clean; failures appear between hours 2 and 3.5 when marginal thermal interfaces or underperforming fans reach their steady-state limit.
The 4-hour baseline is not arbitrary. It matches the time constant of large liquid cooling loops (CDU buffer tanks, rack manifolds) to reach thermal equilibrium, and it gives enough data to distinguish a stable system from one that drifts. For clusters with variable workloads, we run a synthetic load that pins all GPUs to 100% compute and memory simultaneously—a tool like gpu_burn or NVIDIA's DCGM with a matrix multiplication kernel works. Do not use idle or partial loads; they mask problems. If the OEM specifies a longer soak (some require 8 hours for liquid-cooled racks), follow that. Our rule: never shorten the soak below 4 hours, and never skip it for time.
Watch Items: Temperature, Power, and Delta Metrics
The primary watch item is GPU junction temperature (Tj). Log it every 30 seconds from nvidia-smi or the BMC. A pass means Tj never exceeds the OEM maximum (check the spec sheet for your exact GPU, typically 85–95°C) at any point during the soak. A single spike above the limit is an automatic fail—do not retest without fixing the root cause. Second, monitor the delta between GPU inlet air temperature and exhaust air temperature at the rack rear. For air-cooled racks, this delta should stay under 15°C at full load per common rack manufacturer guidelines; a higher delta means recirculation or blocked vents. For liquid-cooled racks, track the coolant supply temperature at the rack manifold and the return temperature from each cold plate. The delta across the loop must remain within the CDU manufacturer's spec (typically 10–15°C). A sudden increase of more than 3°C above the baseline after 2 hours indicates a partial blockage or pump degradation.
Power draw per GPU is equally critical. Each GPU should draw within ±5% of its TDP after the first 30 minutes of soak. A downward drift of more than 5% over the next 3 hours signals thermal throttling—the GPU is reducing clock speed to stay within temperature limits. That is a fail, even if Tj never hits the absolute maximum. Also watch NVSwitch temperature on NVL72 racks: it must stay below its OEM threshold (typically 85°C). If it rises more than 5°C above ambient after 2 hours (verify with your specific switch spec), the liquid cooling path to that switch is compromised. Finally, log memory errors from nvidia-smi's ECC counters. Any correctable or uncorrectable error after the first 30 minutes is a fail—memory errors under load often precede permanent failures.
Pass/Fail Criteria: Objective Thresholds
A pass requires all of the following: (1) No GPU Tj exceeds the OEM maximum at any point during the entire 4-hour soak. (2) Power draw per GPU stays within ±5% of TDP after the first 30 minutes, with no sustained downward trend. (3) NVSwitch temperature stays below its OEM threshold, and its rise above ambient does not exceed 5°C after 2 hours (or as specified by the OEM). (4) Liquid cooling loop delta-T remains within the CDU spec, with no sudden increases. (5) All GPU memory error counters (correctable and uncorrectable) are zero during the final 2 hours. (6) No rack-level thermal alarms (from BMC or CDU) trigger during the soak. If any of these fail, the system fails—do not accept a partial pass.
A fail means stop the burn-in, identify the root cause, and remediate before retesting. Common fixes: reseat a cold plate, replace thermal paste, clean a fan filter, or adjust CDU flow rate. Do not simply reset counters and rerun—that masks the problem. For clusters with multiple racks, each rack must pass individually; a single failing rack can cause a cascade of thermal issues in adjacent racks due to hot exhaust recirculation. Document all pass/fail decisions with timestamped logs from the BMC and nvidia-smi. Leviathan Systems requires a signed-off report for every rack before it enters production.
Common Failure Modes: What Goes Wrong in the Field
The most frequent failure during thermal burn-in is thermal throttling without a Tj exceedance. A GPU may draw 5–10% less power after 2 hours because its VRM or memory is overheating, even though the die itself stays below the limit. This shows up as a power drift in the logs. We have seen this on H100 clusters where a single fan in the GPU tray failed at 70% duty cycle—the BMC didn't flag it because the fan was still spinning, but airflow dropped enough to cause throttling. Another common failure is a blocked cold plate on liquid-cooled NVL72 racks. A piece of debris or a kinked hose can reduce flow to one GPU, causing its Tj to rise 10°C above its neighbors within 30 minutes. The delta-T across the loop may only increase by 1–2°C, so you must monitor per-GPU temperatures, not just loop averages.
A third failure mode is memory errors under thermal stress. GPUs that pass idle memory tests often develop correctable errors after 2 hours of full load when the memory controller heats up. These errors are logged by nvidia-smi's ECC counters but may not trigger an alarm. If you do not check the counters manually, you will miss them. We also see failures from loose power cables: a GPU that draws 700W under load can heat its power connector enough to cause a voltage drop, which the GPU compensates for by drawing more current, creating a thermal runaway. This shows as a gradual power increase over the soak, not a decrease. Finally, liquid cooling loop issues like air pockets or pump cavitation can cause intermittent temperature spikes that appear only after 3 hours when the coolant reaches its hottest point. Always run the full 4 hours.
Tooling and Data Collection Requirements
You need a calibrated temperature logging system that records GPU Tj, NVSwitch temperature, inlet/outlet air temperature, coolant supply/return temperature, and GPU power draw at intervals no longer than 30 seconds. Use the BMC's sensor data or a dedicated monitoring tool that pulls from nvidia-smi and the CDU's API. Do not rely on manual readings—they miss transient spikes. For liquid-cooled racks, install flow meters on each cold plate or at the rack manifold to detect blockages. A flow drop of more than 10% from baseline is a fail, even if temperatures look normal.
For data collection, create a baseline at the start of the soak: record ambient temperature, coolant temperature, and GPU idle temperatures. Then log every 30 seconds for the full 4 hours. At the end, generate a report showing min, max, and average for each metric, plus a graph of Tj over time. The pass/fail decision is based on the thresholds above, not on subjective judgment. Leviathan Systems uses a standardized script that automates this logging and flags any threshold exceedance in real time. If you do not have such a script, write one before starting burn-in—manual logging is error-prone and will miss the subtle drifts that indicate marginal hardware.
Post-Burn-In Validation: What to Do After a Pass
After a rack passes thermal burn-in, do not immediately declare it production-ready. Run a 30-minute idle cooldown and verify that all temperatures return to within 5°C of the pre-burn-in baseline. A GPU that stays 10°C above ambient after cooldown may have a stuck fan or a blocked cold plate that only manifests at low load. Then run a connectivity test for the scale-out network (InfiniBand or Ethernet) to ensure no fiber or MPO links were disturbed during the burn-in. The NVLink copper spine inside the rack is separate and should have been validated during assembly; burn-in does not test it.
Finally, update the rack's asset management system with the burn-in results, including the pass/fail status, the log file location, and any remediation actions taken. For clusters with multiple racks, stagger the burn-in so that no two adjacent racks run simultaneously—hot exhaust from one rack can affect the inlet temperature of its neighbor, causing false failures. Leviathan Systems schedules burn-in for no more than 50% of racks in a row at once. This ensures ambient conditions remain stable and the pass/fail criteria are valid.
Standards referenced: ASHRAE TC 9.9 (thermal guidelines for data center environments, including recommended inlet air temperature ranges) · NVIDIA GPU thermal specifications (Tj max, TDP, NVSwitch temperature limits—available on the NVIDIA documentation portal) · CDU manufacturer specifications for delta-T, flow rate, and coolant temperature range (varies by model and must be obtained from the equipment manual)
Frequently asked_
Can I shorten the burn-in to 2 hours if the cluster is small or the GPUs are new?
No. Two hours is insufficient to detect thermal creep, which often appears between hours 2 and 3.5. New GPUs can have defective thermal paste or cold plate mounting that only shows under sustained load. The 4-hour minimum is a field-proven standard; shortening it risks deploying marginal hardware that will fail under production load, causing costly downtime. Always run the full soak.
What should I do if a GPU's Tj spikes above the limit for just a few seconds?
That is a fail. A transient spike indicates a thermal interface issue (e.g., a bubble in the thermal paste, a loose cold plate, or a fan that momentarily dropped speed). Do not retest without investigating. Reseat the cold plate or replace the thermal paste, then rerun the full 4-hour soak. Ignoring a single spike can lead to a permanent failure weeks later.
How do I distinguish thermal throttling from a genuine power supply issue?
Thermal throttling shows as a gradual power draw decrease over time (e.g., 5% drop over 2 hours) while Tj remains below the limit. A power supply issue typically causes a sudden power drop or fluctuation, often accompanied by voltage warnings in the BMC logs. If power draw drops abruptly (more than 10% in 5 minutes), check the power cables and PSU first. If it drifts down slowly, investigate cooling.
Do I need to monitor memory errors during burn-in, or is temperature enough?
Temperature alone is not enough. Memory errors under thermal stress are a common failure mode that does not correlate perfectly with Tj. A GPU may run at 80°C (well below the limit) but still develop correctable ECC errors because its memory controller is failing. Always check nvidia-smi's ECC counters at the start and end of the soak, and log any non-zero values. Zero errors in the final 2 hours is a pass requirement.
What if the liquid cooling loop delta-T stays within spec but one GPU runs 10°C hotter than its neighbors?
That is a fail. The delta-T across the loop is a coarse metric that can mask a single blocked cold plate. Per-GPU temperature monitoring is mandatory. If one GPU is 10°C hotter than the average of its peers under the same load, it has a thermal interface problem—likely a cold plate that is not making full contact or a flow restriction. Investigate and fix before retesting.