Commissioning_
NCCL All-Reduce Validation as a Cluster Acceptance Gate
Shows deployment engineers how to run and interpret NCCL all-reduce tests on the scale-out fabric as the final objective gate before cluster acceptance, separating intra-rack NVLink copper paths from inter-rack InfiniBand or Ethernet links.
Key facts
- NCCL all-reduce measures sustained bandwidth and latency across the scale-out network only; it does not exercise the copper NVLink spine inside a rack.
- MPO trunk cables for the compute fabric are factory-terminated; field work consists of routing, cleaning, inspection with a fiberscope, and end-to-end testing with an MPO continuity tester or OTDR.
- All-reduce performance is limited by the weakest link in the switched fabric, so tests must run at full job size matching the production allocation.
- NVIDIA provides nccl-tests binaries that report busbw and algbw; acceptance requires both values to meet the OEM-published targets for the installed NIC and switch generation.
- InfiniBand or RoCE links must be brought up and verified with ibdiagnet or equivalent before NCCL tests begin, because a single down port will collapse collective bandwidth.
- Factory-polished MPO connectors require specific cleaning and inspection sequences per the connector vendor; residue on one ferrule end-face can degrade effective bandwidth on a high-speed link.
Distinguishing NVLink from Scale-Out Fabric in Acceptance Testing
NCCL all-reduce runs over the inter-rack network fabric, not the copper NVLink backplane. In an NVL72-class rack the GPUs exchange data inside the rack through the NVLink spine; any NCCL job that stays within one rack will therefore show full line-rate regardless of the external cabling. The acceptance gate must therefore force the test across multiple racks so that the switched InfiniBand or Ethernet fabric carries the traffic.
Engineers first confirm rack-level NVLink topology with nvidia-smi, then launch the NCCL test with a node list that spans the full fabric under test. This ordering prevents a clean intra-rack result from masking a defective leaf or spine switch port.
Pre-Test Checks on the MPO and Switch Fabric
Before any collective test starts, every MPO trunk that carries the scale-out network must be inspected and cleaned. Use a calibrated fiberscope on both ends of each MPO connector; any scratch or particle larger than the core diameter fails the link. After cleaning, run an MPO continuity tester end-to-end and an OTDR on longer trunks to locate bends or macro-bends introduced during rack installation.
Next bring up the switch fabric with the vendor’s diagnostic suite. Confirm that every port in the route between test nodes shows link-up at the expected speed and that no error counters are incrementing. Only after these steps are complete does the fabric become a candidate for NCCL validation.
Running NCCL All-Reduce at Production Scale
Execute nccl-tests with the all_reduce_perf binary using the exact message sizes and iteration counts specified in the cluster acceptance plan. Set the number of processes equal to the number of GPUs that will participate in production jobs; smaller rings hide contention that appears only at full scale.
Capture both algorithmic bandwidth and bus bandwidth on every run. Record the minimum, median, and maximum values across at least three consecutive executions. Store the raw output together with the exact node list, NCCL version, and driver versions so that later regressions can be compared directly.
Interpreting Results Against OEM Targets
Compare the measured bus bandwidth against the OEM-published target for the installed ConnectX generation and switch ASIC. A result that falls more than a few percent below target indicates either a mis-cabled port, an unclean MPO, or a switch buffer configuration issue. Latency outliers at small message sizes point to congestion or suboptimal routing rather than simple link errors.
If any run fails the threshold, quarantine the affected rack pair, re-inspect the MPO path between them, and retest only after the fabric diagnostic passes again. Do not accept the cluster until three consecutive full-scale runs meet the published numbers.
Common Failure Modes Encountered in the Field
The most frequent cause of low all-reduce bandwidth is a single dirty or damaged MPO connector on a high-radix switch port. Because the test aggregates across hundreds of links, one degraded connector throttles the entire collective. Technicians catch this by comparing per-port error counters before and after the NCCL run; the port with rising CRC or symbol errors is the culprit.
Another recurring issue is an incomplete or mismatched subnet manager configuration that leaves one leaf switch operating at reduced speed. This appears as a step-function drop in bandwidth once the job size exceeds the number of GPUs on the misconfigured switch. Running ibdiagnet immediately before NCCL eliminates this class of failure.
Documenting the Gate for Handover
Leviathan Systems records the full NCCL output, fiber inspection images, and switch fabric logs in the commissioning package. The acceptance certificate references the exact test parameters and the three passing runs. Any deviation from the OEM target is noted with the corrective action taken.
This package becomes the baseline for future maintenance; subsequent site expansions rerun the identical NCCL command set against the same node list so that performance drift is immediately visible.
Standards referenced: InfiniBand Architecture Specification · IEEE 802.3 Ethernet standards for 400 Gb/s and 800 Gb/s ports
Frequently asked_
How many racks must participate in the NCCL all-reduce acceptance run?
The test must span at least two racks and ideally the full production allocation size. A single-rack run only exercises NVLink copper and cannot validate the scale-out fabric. The node list is taken directly from the cluster scheduler configuration that will be used in production.
What tool confirms an MPO trunk is clean enough before NCCL testing?
A fiberscope with 400x or higher magnification is used on every MPO end-face. Any contamination or defect visible under the scope fails the connector; it is cleaned with the OEM-approved kit and re-inspected until it passes. An MPO continuity tester then verifies pin-out before the link is placed in service.
Does a passing NCCL result guarantee every individual link is good?
No. NCCL reports aggregate bandwidth; a single marginal link can be masked if enough parallel paths exist. Therefore the switch port error counters and OTDR traces are reviewed in parallel with the NCCL numbers. Any port showing errors is repaired before the cluster is accepted.
When should Leviathan Systems rerun the NCCL gate after initial acceptance?
Rerun the identical test after any fabric change—new rack addition, MPO trunk replacement, or firmware update on the switches. The new results are compared against the baseline recorded at first acceptance; any sustained drop triggers a full fabric diagnostic.
What message size range is used for the acceptance all-reduce test?
The acceptance plan specifies the same message sizes that production jobs will use, typically from 1 MiB to the largest all-reduce buffer the workload employs. Both small-message latency and large-message bandwidth are recorded so that the fabric is validated across the full operating range.