Commissioning_
Commissioning Levels L1–L5 for GPU Data Centers, Explained
Defines L1 through L5 commissioning levels for GPU racks with liquid cooling and scale-out fabrics, specifying the exact verifications performed at each stage for AI data center deployments.
Key facts
- L1 covers factory acceptance testing of individual racks and components before shipment per OEM procedures.
- MPO trunk cables for InfiniBand or Ethernet scale-out networks are factory-terminated; field work consists of routing, cleaning, inspection, and end-face testing.
- NVLink connectivity between GPUs occurs over the internal copper backplane within each rack and is verified through nvidia-smi after rack power-up.
- Liquid cooling loops require sequential pressure testing, flow verification, and leak checks against the rack OEM specification before electrical energization.
- L5 integrated systems testing exercises the full rack population under sustained AI training workloads while monitoring power, thermals, and fabric stability.
- TIA-942 provides the baseline for data center infrastructure tiers that commissioning levels map against for availability targets.
- Common field failures include contaminated MPO end faces causing insertion loss above link budget and reversed liquid cooling quick-disconnects that block flow.
L1 Factory Acceptance Testing
L1 verifies that each GPU rack, power shelf, and liquid cooling manifold meets the OEM build sheet before the equipment leaves the integration facility. Technicians run full power-on self-tests, confirm GPU count and firmware revisions, and exercise the internal NVLink domain through the copper spine. Liquid cooling paths receive initial pressure holds and flow measurements recorded against the rack-specific acceptance criteria.
Documentation from L1 travels with the rack and becomes the baseline for all subsequent site checks. Any deviation found later at site is traced back to this record to isolate transport damage from assembly issues.
L2 Site Receipt and Pre-Installation Checks
L2 begins when racks arrive on site. Crews inspect shipping damage, verify serial numbers against the bill of materials, and re-inspect every MPO connector end face with a calibrated scope before any patching occurs. Liquid cooling connections are visually checked for proper quick-disconnect orientation and dust caps are only removed immediately before mating.
These steps prevent contamination from propagating into the optical fabric or cooling loops. Racks are positioned and anchored but remain electrically isolated until L2 sign-off.
L3 Component-Level Functional Testing
L3 energizes each rack subsystem independently. Power distribution units are brought online first, followed by GPU node power-up to confirm nvidia-smi reports all devices and internal NVLink status. Liquid cooling pumps and CDUs are started separately to verify flow rates and differential pressure without GPU heat load.
Network leaf switches are cabled to the MPO trunks and tested for continuity and polarity using an MPO continuity tester. No workload traffic runs yet; the objective is to prove each domain operates within its own specification.
L4 Integrated Rack and Fabric Testing
L4 combines the rack subsystems under coordinated control. Full rack power is applied while the liquid cooling loop maintains set-point temperatures across all GPUs. Scale-out InfiniBand or Ethernet links between racks are validated with OTDR traces and insertion-loss measurements on every MPO patch cord.
Traffic generators exercise the fabric at line rate while monitoring for CRC errors and link flaps. Liquid cooling performance is logged under increasing GPU utilization to confirm heat removal matches the design curve before any multi-rack AI job is attempted.
L5 Full-Scale Systems and Workload Validation
L5 runs representative AI training jobs across the entire cluster for a minimum duration defined by the operator. Metrics collected include sustained power draw per rack, GPU and switch temperatures, fabric latency and bandwidth, and cooling loop stability under variable load. Any node or link that drops out during the test must be isolated, repaired, and retested before the level is accepted.
Leviathan Systems crews witness L5 alongside the operator to transfer operational runbooks and alarm response procedures.
Common Failure Modes Encountered in the Field
MPO end-face contamination remains the leading cause of link failures after L3. Technicians catch this by performing inspection immediately before every mating rather than relying on prior cleaning.
Reversed liquid cooling supply and return quick-disconnects produce zero flow to one or more cold plates; the symptom appears as rapidly rising GPU temperatures during the first L4 ramp. Pressure testing each loop segment before GPU power-up reveals the reversal. NVLink errors reported by nvidia-smi after rack move-in almost always trace to unseated internal copper cables disturbed during transport, not to any fiber plant issue.
Standards referenced: TIA-942 · IEC 61754-7 (MPO interface) · OEM rack liquid cooling specification · TIA-568 series for structured cabling
Frequently asked_
How does L4 differ from L3 for the optical fabric?
L3 confirms individual MPO links pass continuity and basic loss tests. L4 adds multi-rack traffic at line rate, measures end-to-end latency, and verifies that the full fabric remains stable when all racks draw maximum power and generate maximum heat simultaneously.
What specific checks occur on the liquid cooling loop before L4 begins?
Each manifold segment receives a low-pressure leak test, followed by a high-pressure hold per the OEM procedure. Flow is then verified at the design rate with all GPUs powered but idle, confirming no air locks or reversed connections exist before heat load is applied.
Why is internal NVLink never tested over fiber during commissioning?
NVLink between GPUs inside an NVL72-class rack uses the copper backplane supplied with the rack. The fiber and MPO infrastructure carries only the scale-out compute fabric between switches and racks; conflating the two domains leads to incorrect troubleshooting.
At what point are OTDR traces required on the MPO trunks?
OTDR traces are performed during L4 once all patch cords are installed and cleaned. They establish the baseline loss and event map for the entire optical path before any production AI traffic runs across the links.
Who typically signs off on L5 completion?
The operator, the rack OEM representative, and the commissioning crew all sign the L5 report. The document lists every rack that passed sustained workload testing along with any exceptions and their resolutions.