Commissioning_
GPU Commissioning & Acceptance: What to Demand Before You Sign Off
A field-tested checklist of test results, as-built documentation, and acceptance criteria that data-center operators must demand from their deployment crew before signing off on a GPU rack—covering power, cooling, networking, and structural integrity for NVL72-class systems.
Key facts
- NVL72 racks use a copper NVLink spine inside the rack for GPU-to-GPU communication; fiber/MPO carries only the scale-out network (InfiniBand or Ethernet).
- MPO trunk cables are factory-terminated and polished; field work is limited to patching, routing, cleaning, inspection, and testing—never field-crimping.
- A calibrated MPO continuity tester and an OTDR are required to validate fiber links; the relevant standard for MPO end-face inspection is IEC 61300-3-35.
- Liquid cooling loop pressure must hold within the OEM-specified tolerance for the duration specified by the OEM (commonly 30 minutes) after filling, with no measurable drop beyond the OEM threshold.
- GPU power draw per node must be logged during a sustained stress test (typically 30 minutes at full load per OEM guidance) and compared against the OEM spec to catch under- or over-voltage conditions.
- All structured cabling must be routed with bend radii exceeding the manufacturer’s minimum rating (typically 10x the cable diameter for fiber) and secured with proper strain relief at intervals per cabling best practices (commonly 1.5 meters).
- As-built documentation must include rack elevation diagrams, cable run lists with lengths and patch-panel ports, liquid cooling loop schematics, and test result logs for every link.
Power Integrity: Validating Voltage Rails and Load Distribution
Before signing off, demand a full power-rail validation report. Each GPU node must be stress-tested at sustained full load for the duration specified by the OEM (typically 30 minutes) while logging per-node voltage, current, and power draw using a calibrated power analyzer. Compare these logs against the OEM spec—any node drawing more than 5% above its rated TDP (or the OEM’s tolerance) indicates a potential thermal or electrical issue that should be investigated before acceptance. Also verify that the rack-level PDU or busbar is not exceeding its rated capacity; a single overcurrent event can cascade to a full rack shutdown.
Use a calibrated power analyzer at the rack input to confirm total power draw matches the sum of all nodes plus overhead for fans, pumps, and switches. Document the voltage at each PDU outlet under load—voltage drop beyond the OEM tolerance (typically 5% from nominal) suggests undersized cabling or poor connections. Reject any rack where the power distribution unit logs show repeated voltage sags or current spikes during the stress test, as these indicate electrical instability that can cause GPU resets or long-term damage.
Liquid Cooling Loop Integrity: Pressure Hold and Flow Verification
For liquid-cooled racks, the cooling loop must pass a pressure-hold test before any GPU is powered. Fill the loop with the specified coolant, pressurize to the OEM’s recommended test pressure (commonly 1.5 times the operating pressure), and monitor for the OEM-specified hold time (typically 30 minutes). A pressure drop greater than the OEM’s threshold (often 0.1 bar) indicates a leak—do not proceed until the leak is located and repaired. Use a calibrated pressure gauge and log the pressure every minute during the test.
After the pressure hold, verify flow rates at each cold plate or manifold. The flow rate per node must fall within the OEM’s specified range (commonly 1–2 liters per minute for a typical GPU cold plate). Use an inline flow meter or ultrasonic clamp-on meter. If any node shows flow below the minimum, check for kinked tubing, blocked quick-disconnects, or air pockets. Air pockets can cause hot spots and eventual GPU failure—bleed the loop per the OEM procedure and retest. A thermal camera during the pressure hold can also reveal slow leaks by showing evaporative cooling at the leak point.
Network Link Certification: MPO/Fiber and Copper NVLink Spine
Every MPO trunk cable in the scale-out network must be inspected and tested. Use a calibrated MPO continuity tester to verify polarity and fiber count, then an OTDR to measure insertion loss and reflectance on each fiber. The relevant standard is IEC 61300-3-35 for end-face inspection—reject any connector with scratches, pits, or contamination visible under a 200x or 400x microscope. Document the pass/fail status for every link in a cable run list. A single bad fiber can bring down a link, and contamination is the most common root cause.
Do not conflate the fiber network with the NVLink spine. The NVLink spine inside the rack is copper and is tested via the GPU’s firmware (e.g., nvidia-smi shows NVLink status). However, you must still verify that the spine cables are seated correctly and that the retention clips are engaged. A loose copper NVLink cable can cause intermittent GPU-to-GPU bandwidth drops that are hard to diagnose later. Run a fabric-level bandwidth test (e.g., using NCCL or a vendor-provided tool) to confirm all NVLink links are at full speed. This test should run for at least 10 minutes per OEM guidance.
Structured Cabling: Bend Radius, Strain Relief, and Labeling
All fiber and copper cables must be routed with bend radii exceeding the manufacturer’s minimum rating—typically 10 times the cable diameter for fiber and 4 times for copper. Use a bend-radius gauge or a template to check tight bends, especially near patch panels and cable trays. Cables must be secured with Velcro straps (never zip ties) at intervals no greater than the cabling standard recommendation (commonly 1.5 meters) to prevent sagging and strain on connectors. Strain relief bars or cable managers must be used at every patch panel and switch port.
Label every cable at both ends with a unique identifier that matches the as-built documentation. The label should include the source port, destination port, and cable length. Reject any rack where cables are tangled, unlabeled, or routed over sharp edges. Poor cable management leads to accidental disconnections during maintenance and makes troubleshooting nearly impossible. Field experience from Leviathan Systems shows that unlabeled cables add hours to every maintenance event.
Common Failure Modes: What Goes Wrong in the Field and How to Catch It
The most common failure during commissioning is a loose or contaminated MPO connector. Even a single speck of dust on a ferrule can cause high insertion loss or back-reflection, dropping the link. Always clean and inspect every MPO connector before mating—use a one-click cleaner and a 200x microscope. Another frequent issue is incorrect MPO polarity (e.g., Type A vs. Type B) causing a link to fail entirely. Verify polarity with a continuity tester before connecting to active equipment.
Liquid cooling leaks are the second most common failure. They often occur at quick-disconnect couplings or compression fittings that were not fully seated. During the pressure-hold test, use a thermal camera to look for cold spots on the loop—a cold spot may indicate a slow leak where coolant is evaporating. Also check for kinked tubing behind cable trays; a kink can restrict flow and cause a GPU to overheat. Finally, power issues: loose PDU connections or undersized breakers can cause intermittent resets. Log all power events during the stress test and investigate any that exceed the OEM’s threshold for voltage sag or current spike duration. Catching these issues before sign-off saves weeks of troubleshooting later.
As-Built Documentation: What Must Be in the Handoff Package
Demand a complete as-built package before signing off. This must include: rack elevation diagrams showing the physical location of every GPU node, switch, and PDU; a cable run list with unique cable IDs, lengths, source/destination ports, and test results for every link; liquid cooling loop schematics with pipe diameters, valve locations, and flow rates; and a test log for every power, cooling, and network test performed. The documentation should be in a non-proprietary digital format (PDF and editable spreadsheet) and include a revision history.
Without this package, future maintenance and troubleshooting become guesswork. For example, if a GPU fails six months later, the as-built cable run list tells you exactly which MPO trunk to check and where it’s patched. The cooling schematic shows which valve to isolate. The power log shows whether the node was borderline from day one. Reject any handoff that lacks these documents—they are not optional. Operators working with Leviathan Systems consistently require these documents before accepting deployment.
Standards referenced: IEC 61300-3-35 (MPO end-face inspection) · TIA-568.3-D (optical fiber cabling component standard) · OEM-specific pressure-hold test parameters (e.g., 1.5x operating pressure, 30-minute hold)
Frequently asked_
What is the minimum pressure-hold time for a liquid cooling loop?
The typical OEM specification is a pressure hold of at least 30 minutes at 1.5 times the operating pressure. A pressure drop greater than the OEM threshold (commonly 0.1 bar) during that period indicates a leak. Use a calibrated pressure gauge and log readings every minute. If the loop passes, proceed to flow verification.
Do I need to test every fiber in an MPO trunk?
Yes. Every fiber in every MPO trunk must be tested with an OTDR for insertion loss and reflectance, and inspected under a 200x microscope per IEC 61300-3-35. A single bad fiber can bring down a link. Use a calibrated MPO continuity tester first to verify polarity and fiber count, then OTDR each fiber. Document all results in the cable run list.
How do I verify the NVLink spine is working without a full system boot?
You cannot fully verify NVLink without powering the GPUs and running a fabric test (e.g., nvidia-smi topo -m or NCCL bandwidth test). However, you can visually inspect the copper spine cables for proper seating and retention clip engagement. Loose cables cause intermittent drops. Once powered, run a sustained bandwidth test for at least the duration recommended by the OEM (typically 10 minutes) to confirm all links are at full speed.
What should I do if a GPU node draws more power than its rated TDP during stress test?
Log the exact power draw and compare it to the OEM spec. If it exceeds the rated TDP by more than the OEM’s tolerance (commonly 5%), investigate the node’s cooling and voltage regulation. Check for airflow obstructions, liquid cooling flow rate, and PSU voltage. If the issue persists, replace the node before signing off. A node running hot from day one will have a shortened lifespan.
Who typically performs the commissioning tests—the deployment crew or the operator?
The deployment crew (e.g., Leviathan Systems) performs the tests and provides the results in the as-built package. The operator’s team should witness key tests (pressure hold, OTDR, power stress test) and review the documentation before signing off. Never accept a verbal 'it’s all good'—demand the test logs and inspect them yourself.