Commissioning_
Pre-Power Inspection: The Walkdown Before Energizing a GPU Hall
A step-by-step field guide to the pre-power walkdown inspection for a GPU hall, covering every check from rack bonding to MPO trunk continuity, with failure modes and decision criteria that prevent arc flash, data corruption, and costly rework.
Key facts
- A pre-power walkdown must verify rack bonding to earth ground per TIA-607-D, with a resistance value below the OEM-specified maximum (typically less than 0.1 ohm) to prevent ground loops that damage GPU transceivers.
- MPO trunk cables in the scale-out network must be tested with a calibrated MPO continuity tester and an OTDR; a single dirty or damaged ferrule can cause link errors that mimic GPU failure during nvidia-smi diagnostics.
- Liquid cooling loops must be pressure-tested at the OEM-specified hold pressure for a minimum of 30 minutes; a pressure drop exceeding the OEM tolerance indicates a leak that will short electronics if energized.
- GPU-to-GPU NVLink runs over the copper NVLink spine/backplane inside the rack—not over fiber or MPO—so the walkdown must verify backplane seating and torque of all NVLink connectors per the rack OEM's torque spec.
- Power distribution unit (PDU) input voltage and phase rotation must be verified with a phase rotation meter before rack PSUs are inserted; reversed phase rotation can trip main breakers and damage PSUs.
- Structured cabling pathways must maintain a bend radius no less than the cable manufacturer's spec (typically 10x the cable OD for fiber, 4x for copper) to avoid micro-bends that cause intermittent link failures.
- A visual inspection of all rack doors and cable management arms (CMAs) must confirm they close without pinching or stressing any cable; a pinched fiber trunk can cause a complete loss of scale-out connectivity to an entire rack row.
Rack Bonding and Ground Verification
Before any power is applied, every rack must be bonded to the facility earth ground per TIA-607-D. Use a calibrated micro-ohmmeter to measure resistance from the rack's main bonding jumper to the ground bus bar. The reading must be below the OEM-specified maximum—typically less than 0.1 ohm—because higher resistance creates ground loops that inject noise into GPU transceivers and can cause intermittent link errors that are nearly impossible to debug post-energization.
Check that all rack bonding conductors are sized per the rack OEM's installation manual and that lugs are torqued to the spec. A loose lug can arc under load, creating a fire risk. Also verify that the rack's ground stud is not painted over—paint acts as an insulator. If the rack has sliding rails, confirm the bonding strap is intact and not kinked. A single ungrounded rack in a hall of hundreds can cause a ground potential rise that damages every GPU in that row.
Power Distribution and Phase Rotation Check
With all rack PSUs removed or in the off position, use a phase rotation meter on each PDU input to confirm the phase sequence matches the facility's labeling (typically A-B-C clockwise). Reversed phase rotation will cause three-phase PSUs to draw current incorrectly, tripping upstream breakers or damaging the PSU's input rectifier. This is a common failure mode when PDUs are installed by different crews on different shifts.
Next, measure voltage between each phase and neutral, and between phases, at the PDU output receptacles. Compare to the PSU input voltage range printed on the PSU label. If any receptacle is outside spec, do not insert a PSU until the facility electrician corrects the feed. Also verify that the PDU's main breaker is in the off position and locked out/tagged out per your site's LOTO procedure. This prevents accidental energization while you are still working in the rack.
Liquid Cooling Loop Integrity Test
For liquid-cooled GPU racks, the cooling loop must be pressure-tested before any power is applied. Close all manual isolation valves at the rack manifold, then pressurize the loop with dry nitrogen or the OEM-specified test fluid to the hold pressure stated in the rack's installation manual. Hold for a minimum of 30 minutes. A pressure drop exceeding the OEM tolerance—typically a few psi—indicates a leak at a quick-connect fitting, hose barb, or cold plate seal.
If a leak is detected, depressurize, locate the leak using a soap-and-water bubble test or electronic leak detector, and re-torque the fitting or replace the component. Never attempt to energize a rack with a known leak: coolant is conductive and will short GPU boards, causing catastrophic failure. After the pressure test passes, open the isolation valves and verify flow through each GPU cold plate using the rack's flow indicator or manifold sight glass. No flow means a blocked line or closed valve—fix before proceeding.
NVLink Backplane and GPU Seating Verification
The NVLink spine/backplane inside the rack carries all GPU-to-GPU traffic. Before power-on, visually inspect that each GPU module is fully seated into its backplane slot. The backplane connector should be flush with the GPU module's edge connector—no gap. Use a flashlight to check for bent pins in the backplane socket. A single bent pin can cause an NVLink link to fail, which nvidia-smi will report as a GPU error, wasting hours of troubleshooting.
Torque each GPU module's retention screws to the rack OEM's spec using a calibrated torque driver. Under-torqued screws allow vibration to unseat the module over time; over-torqued screws can crack the PCB or backplane. Also verify that the NVLink bridge cables (if used in non-backplane designs) are routed with proper bend radius and fully inserted into their sockets. A loose bridge cable will cause intermittent NVLink errors that are hard to reproduce.
Scale-Out Network MPO Trunk Inspection and Testing
The scale-out network (InfiniBand or Ethernet) uses MPO trunk cables between rack top-of-rack switches and leaf/spine switches. Every MPO trunk must be inspected with a fiber inspection scope at both ends before insertion. A single dust particle on a ferrule can cause a link to flap or fail, which the GPU driver may misinterpret as a hardware fault. Clean any dirty ferrules with a dry clicker cleaner or lint-free wipes and isopropyl alcohol per the connector manufacturer's instructions.
After cleaning, test each trunk with a calibrated MPO continuity tester to verify polarity (Type A, B, or C per TIA-568.3) and then with an OTDR to measure end-to-end loss and reflectance. The loss must be below the OEM-specified maximum for the link length and connector count. If a trunk fails, do not patch it—replace it. Field-terminated MPO ferrules are not used in this environment; all trunks are factory-terminated and polished. Also verify that trunk cables are routed in cable trays with proper bend radius (typically 10x the cable OD) and secured with Velcro ties—never zip ties—to avoid micro-bends.
Structured Cabling Pathway and Cable Management Check
All copper and fiber patch cables must be routed through cable management arms (CMAs) or horizontal cable managers without exceeding the cable's minimum bend radius. For fiber, this is typically 10x the cable OD; for copper, 4x. A cable bent tighter than spec will have micro-bends that cause signal loss or intermittent errors. Use a bend-radius gauge to verify tight bends—do not guess.
Check that all cable ties are Velcro and hand-tight—never zip ties, which can crush fiber and cause permanent loss. Also verify that rack doors close fully without pinching any cable. A pinched fiber trunk can cause a complete loss of scale-out connectivity to an entire rack row. Finally, label both ends of every patch cable per your site's labeling standard (e.g., TIA-606-B). Unlabeled cables are the leading cause of commissioning delays when a link fails and no one knows which cable to trace.
Common Failure Modes and How to Catch Them
The most frequent failure during pre-power walkdowns is a ground bond that tests high due to a painted rack stud or loose lug. Always scrape paint off the bonding surface and re-torque. Another common issue is reversed phase rotation on PDUs, which is silent until power is applied and a PSU fails. Always use a phase rotation meter, not just a voltmeter.
MPO trunk failures are often caused by dirty ferrules that were cleaned but re-contaminated by touching a surface. Always inspect after cleaning, and use dust caps on unmated connectors. Liquid cooling leaks are most common at quick-connect fittings that were not fully seated—listen for a hiss during pressurization. Finally, NVLink backplane pins can be bent by a GPU module inserted at an angle. Always insert modules straight and check for resistance—if it doesn't slide in easily, stop and inspect the socket.
Standards referenced: TIA-607-D (Bonding and Grounding for Telecommunications) · TIA-568.3 (Optical Fiber Cabling Components Standard) · TIA-606-B (Administration Standard for Telecommunications Infrastructure) · IEC 60364 (Low-Voltage Electrical Installations, grounding requirements)
Frequently asked_
What is the most common mistake during pre-power walkdown that causes a delay in commissioning?
The most common mistake is skipping the phase rotation check on PDUs. Crews often assume the facility electrician got it right, but reversed phase rotation is surprisingly common when multiple contractors work on the same feed. It's silent until you try to power on a PSU, at which point it trips the main breaker and can damage the PSU. Always use a phase rotation meter before inserting any PSU.
How do I know if an MPO trunk is good without expensive test equipment?
You cannot fully verify an MPO trunk without a calibrated MPO continuity tester and an OTDR. A visual inspection with a fiber scope is necessary but not sufficient—it only checks for dirt or scratches on the ferrule, not for internal fiber breaks or high loss. At minimum, use a continuity tester to check polarity and a light source and power meter to measure end-to-end loss. Leviathan Systems always carries these tools on every walkdown.
Can I energize a rack if the liquid cooling loop pressure test shows a small drop?
No. Any pressure drop exceeding the OEM tolerance indicates a leak. Even a tiny leak can drip coolant onto a GPU board during operation, causing a short that destroys the board and potentially the entire rack. Depressurize, find the leak with a bubble test, fix it, and re-test. Never skip this step—it's cheaper to delay commissioning by an hour than to replace a rack of GPUs.
What should I do if I find a bent pin in an NVLink backplane socket?
Stop immediately and do not insert a GPU module. A bent pin can short adjacent pins, causing a cascade failure that damages the backplane and multiple GPUs. Use a magnifying glass and a fine-tipped tool (like a dental pick) to gently straighten the pin. If the pin breaks or cannot be straightened, the backplane must be replaced. Document the issue and escalate to the rack OEM's support team.
How tight should I torque GPU module retention screws?
Use the torque spec printed in the rack OEM's installation manual—never guess. Over-torquing can crack the GPU PCB or backplane; under-torquing allows vibration to unseat the module. A calibrated torque driver is required. If you don't have one, do not proceed until you do. Leviathan Systems uses torque drivers calibrated annually to ensure accuracy.