Writing an Acceptance Test Plan (ATP) for a GPU Cluster

Scope boundaries and document prerequisites

The ATP must list every rack serial number, the exact GPU model population, and the copper NVLink spine configuration before any measurement begins. It also records the scale-out switch ports and the MPO trunk identifiers that will connect this rack to the rest of the fabric.

Prerequisites include completed rack alignment to the raised floor, verified torque on all busbar connections, and delivery of the facility single-line diagram showing the intended feed path. No test proceeds until these items carry dated sign-off from both the mechanical and electrical leads.

The document states explicitly that internal copper NVLink links are excluded from fiber test requirements and that MPO work addresses only the InfiniBand or Ethernet compute fabric.

Mechanical and structural inspection sequence

Begin with a visual check of every rack rail, shelf, and cable management arm against the OEM rack elevation drawing. Measure and record clearance between the copper NVLink spine and any adjacent liquid cooling manifold to confirm no contact points exist.

Next verify that all blanking panels are installed in unoccupied GPU slots and that rear door perforations align with the cold-aisle containment seals. Any deviation requires correction and re-inspection before the ATP advances.

Document the position of each PDU and the routing of its input whips so that future maintenance can isolate a single rack without disturbing neighboring units.

Power distribution and grounding verification

With all PDU breakers locked open, measure continuity from each rack ground bar to the facility grounding electrode system using a low-resistance ohmmeter. Record the value at the rack ground bar and again at the nearest under-floor ground grid point.

Inspect every power whip for proper torque marks on the compression lugs and confirm phase rotation matches the facility standard using a rotation meter. Any reversal or high-resistance reading stops the ATP until corrected.

The plan requires photographs of each torque wrench setting and the final ground resistance reading before the rack is released for the next stage.

Liquid cooling loop integrity checks

Pressurize each manifold segment to the OEM-specified test pressure and monitor for decay over the required hold period. Walk the entire loop with a leak detector and inspect every quick-disconnect and barb fitting under illumination.

Record inlet and outlet temperatures of the facility chilled water supply at the rack isolation valves to establish baseline delta-T before any heat load is applied. Confirm that drain valves are closed and capped.

Any pressure loss or visible moisture requires isolation of the affected segment, repair, and full retest before the ATP continues.

Copper NVLink and scale-out fiber separation

Confirm continuity across every copper NVLink lane on the internal backplane using the rack management controller diagnostic port while breakers remain open. This test verifies seating of the GPU trays against the spine but does not involve external fiber.

For the scale-out network, inspect each MPO connector end-face with a calibrated microscope, clean when required, and test insertion loss with a calibrated MPO continuity tester or OTDR. Record pass/fail against the link budget defined in the facility design package.

The ATP explicitly states that a failed MPO strand does not affect the internal NVLink copper domain and must be resolved through patching or trunk replacement before rack power-up.

Common failure modes encountered in the field

The most frequent stoppage occurs when an MPO connector is cross-patched between two different fabric planes, producing an OTDR trace that shows the correct length but fails continuity at the far end. This is caught by labeling each MPO leg with both rack and switch port identifiers before the first test.

Another recurring issue is residual pressure in a cooling segment caused by an upstream valve left partially open; the decay test then fails even though no leak exists. The remedy is to isolate the rack valves completely and re-zero the gauge before the timed hold begins.

Ground loops created by accidental contact between the rack frame and an uninsulated busbar joint produce unexpectedly low resistance readings that later cause GFCI trips. These are identified by repeating the ground measurement after each mechanical adjustment.

Sign-off gates before energization

Every section of the ATP must carry dated initials from the field lead and the customer commissioning engineer. No section may be marked complete if any single test point remains outside tolerance. Leviathan Systems personnel archive the completed document with rack serial numbers only after all gates pass.

Once all mechanical, power, cooling, and fiber items are accepted, the next rack is released for the same sequence. Only after the full row or suite passes does the plan authorize closure of the first PDU breaker.