Testing_
NCCL Bandwidth Validation: Proving a GPU Fabric Before Production
A field engineer’s guide to running NCCL bandwidth tests on a deployed GPU cluster, interpreting results, and diagnosing fabric faults before production workloads begin.
Key facts
- NCCL (NVIDIA Collective Communications Library) tests measure inter-GPU bandwidth and latency using collective operations like all-reduce, all-gather, and broadcast.
- In a healthy NVL72 rack, NVLink bandwidth between GPUs inside the rack is typically 900 GB/s per GPU (bidirectional), while InfiniBand NDR400 links deliver 400 Gb/s per port.
- NCCL bandwidth should scale linearly with the number of GPUs in a single all-reduce until the fabric’s bisection bandwidth is saturated; a plateau or drop indicates a bottleneck.
- The most common failure mode in field deployments is a marginal MPO connector (dust, damage, or misalignment) causing link errors that reduce effective bandwidth by 10–50% without a full link drop.
- NCCL tests must be run with the correct environment variables (e.g., NCCL_IB_DISABLE=0 for InfiniBand, NCCL_NET_GDR_LEVEL=5 for GPUDirect RDMA) to reflect production settings.
- A failing NCCL test (bandwidth < 80% of expected) should trigger a systematic check: link status (ibstatus), port counters (ibstat), cable inspection, and OTDR trace on fiber links.
- NCCL bandwidth validation is not a substitute for fabric-level stress tests (e.g., HPC Challenge, MPI benchmarks) but is the fastest way to confirm GPU-to-GPU paths are healthy.
What NCCL Tests Actually Measure
NCCL tests (nccl-tests) are microbenchmarks that measure GPU-to-GPU communication performance using collective operations. The most common is all-reduce, which sums data across all GPUs and scatters the result. Other operations include all-gather, reduce-scatter, broadcast, and reduce. Each test reports two metrics: bus bandwidth (total data moved per second across all links) and algorithm bandwidth (data moved per second per GPU, accounting for the operation’s inherent inefficiency).
For production clusters, bus bandwidth on the full GPU count is the key metric. In a healthy NVL72 rack with 72 GPUs connected via NVLink, expected bus bandwidth for all-reduce with 256-MB messages is typically around 900 GB/s per GPU (bidirectional). The test also reports latency for small messages (e.g., 8 bytes), which should be in the low microseconds for NVLink and tens of microseconds for InfiniBand. Establish a baseline from a known-good rack; any deviation greater than 10% signals a fabric fault.
Expected Bandwidth Scaling and How to Verify It
NCCL bandwidth should scale linearly with the number of GPUs until the fabric’s bisection bandwidth is reached. Within a single NVL72 rack, NVLink provides full bisection, so scaling from 8 to 72 GPUs should be near-linear. Once you cross rack boundaries (e.g., 144 GPUs across two racks), the InfiniBand fabric becomes the bottleneck, and scaling becomes sub-linear. The exact knee point depends on the number of uplinks per leaf switch and the network topology (e.g., fat-tree, dragonfly).
To verify, run nccl-tests with increasing GPU counts: all_reduce_perf -b 8 -e 256M -f 2 -g 8, then 16, 32, 64, and 128. Plot bus bandwidth versus GPU count. A smooth curve indicates a healthy fabric; a sudden drop or plateau at a specific count points to a link or switch issue at that scale. Also vary message sizes: small messages (1 KB) stress latency, large (256 MB) stress bandwidth. If large-message is low but small-message latency is fine, the problem lies in the network fabric (cables, switches, NICs) rather than NVLink.
Setting Up the Test Environment Correctly
Before running nccl-tests, ensure environment variables match the production configuration. For InfiniBand clusters, set NCCL_IB_DISABLE=0, NCCL_NET_GDR_LEVEL=5 (GPUDirect RDMA), and NCCL_IB_HCA (to specify the host channel adapter). For Ethernet fabrics, use NCCL_NET=Socket or NCCL_NET=IB with appropriate flags. Enable NCCL_DEBUG=INFO to get per-rank logs that show which GPU or link is failing. Run the test via MPI: mpirun -np <total_gpus> -hostfile hosts all_reduce_perf -b 8 -e 256M -f 2.
Always run on an idle cluster. Verify GPUs are visible (nvidia-smi topo -m) and NVLink is active (nvidia-smi nvlink -s). If any GPU shows NVLink errors, address them first—NCCL tests will degrade or fail. For multi-rack tests, confirm the InfiniBand subnet manager is running and all switches are in a single partition. Check partition membership with ibswitches and ibhosts; a mismatch causes NCCL to hang or produce extremely low bandwidth.
Common Failure Modes and Diagnostic Steps
The most frequent failure in field deployments is a marginal optical link: an MPO connector with dust, a damaged ferrule, or a bend radius violation that causes bit errors. NCCL shows bandwidth 10–50% below expected, but the link may report as up (ibstatus shows LinkUp). Check port counters with ibstat: look for SymbolErrorCounter, LinkErrorRecovery, and LinkDowned. Even a single bad fiber in a 12-fiber MPO trunk reduces effective bandwidth by at least 1/12 (8.3%) plus retransmission overhead. Use a fiber optic continuity tester with appropriate launch conditions and an OTDR to locate the fault. Inspect connector end-faces with a 200x or 400x microscope; clean with a dry-click cleaner or isopropyl alcohol and lint-free wipes.
Another common failure is a misconfigured partition key or subnet manager. If NCCL hangs or shows bandwidth below 10% of expected, check that all InfiniBand ports are in the same partition. Also verify GPUDirect RDMA: reset NVLink counters (nvidia-smi nvlink -g 0 -r), run a small NCCL test, and check for errors. NCCL WARN messages about net plugin timeouts or IB timeouts point to network stack issues, not GPU faults. For persistent problems, run ibdiagnet to scan the fabric for port errors and link flapping.
Interpreting Failing Results: What to Check Next
If NCCL bandwidth is below 80% of expected, follow a systematic checklist. First, confirm test parameters: message size, GPU count, environment variables, and MPI hostfile. Then inspect all MPO connectors with a fiber microscope—scratches, pits, or contamination are common. Clean and re-test. If the issue persists, run an OTDR trace on each fiber in the trunk to identify breaks or high-loss splices. Replace any cable with loss above the OEM spec (typically 0.5 dB per connector pair).
Next, check switch ports: log into the InfiniBand switch (ibdiagnet, ibnetdiscover) and look for CRC errors or link flapping. If a specific port shows errors, reseat the transceiver and cable. If the problem follows the cable, replace it. If it follows the port, the switch module may need replacement. Finally, run a fabric-wide stress test (e.g., HPC Challenge) to isolate whether the issue is NCCL-specific or broader. Document all findings in a deployment report for the operator.
When to Escalate Beyond NCCL Tests
NCCL tests are a fast sanity check but do not cover all failure modes. If NCCL bandwidth is acceptable but production training (e.g., LLM) is slow, the issue may lie in storage I/O, CPU memory bandwidth, or PCIe errors. Run fio for storage, stream for memory bandwidth, and nvidia-smi pci -g for PCIe errors. Also check for thermal throttling: nvidia-smi -q -d TEMPERATURE shows GPU temperatures; if any GPU exceeds the OEM spec (typically 85°C), it will throttle and reduce performance.
If hardware tests pass but NCCL results vary inconsistently, the problem may be software: a kernel driver mismatch, firmware conflict, or misconfigured NCCL topology file. Verify driver, firmware, and NCCL library versions are compatible per NVIDIA documentation. In rare cases, a GPU with a manufacturing defect (e.g., a bad HBM stack) passes basic tests but fails under sustained load. Run a memory stress test (e.g., memtestG80) on suspect GPUs. Leviathan Systems has encountered such defects in the field and recommends a 24-hour burn-in test before production sign-off.
Standards referenced: NVIDIA NCCL documentation (nccl-tests user guide) · InfiniBand Architecture Specification (vol. 1, section on link error counters) · TIA-568.3-D (optical fiber cabling and connector inspection criteria) · IEC 61300-3-35 (fiber optic connector end-face inspection and cleaning)
Frequently asked_
What is the expected NCCL all-reduce bandwidth for a single NVL72 rack with 72 GPUs?
For a single NVL72 rack, expected bus bandwidth for all-reduce with 256-MB messages is typically around 900 GB/s per GPU (bidirectional NVLink). Aggregate bandwidth across all 72 GPUs is approximately 64 TB/s. Actual numbers depend on GPU model (H100 vs. B200) and NVLink generation. Always compare against the OEM spec for your specific hardware. If you see less than 80% of that value, investigate the fabric.
Why does my NCCL test show low bandwidth even though all links are up?
A link being up does not mean it is error-free. Bit errors cause retransmissions that reduce effective bandwidth without dropping the link. Check InfiniBand port counters (ibstat) for SymbolErrorCounter, LinkErrorRecovery, or LinkDowned. Also inspect MPO connectors with a fiber microscope—dust or scratches are the most common cause. Clean and re-test. If the issue persists, run an OTDR trace to find high-loss splices or bends.
How do I run NCCL tests across multiple racks?
Use MPI with a hostfile listing all nodes. For example: mpirun -np <total_gpus> -hostfile hosts all_reduce_perf -b 8 -e 256M -f 2. Ensure the InfiniBand subnet manager is configured to route between racks and that all switches are in the same partition. Set NCCL_IB_DISABLE=0 and NCCL_NET_GDR_LEVEL=5. Start with a small GPU count (e.g., 8 per rack) and scale up to verify cross-rack bandwidth.
What should I do if NCCL tests pass but production training is slow?
NCCL tests only measure GPU-to-GPU communication. Slow production training can be caused by storage I/O bottlenecks, CPU memory bandwidth limits, or PCIe errors. Run fio for storage, stream for memory bandwidth, and nvidia-smi pci -g for PCIe errors. Also check GPU temperatures for thermal throttling. If all hardware tests pass, verify software versions (driver, firmware, NCCL) are compatible per NVIDIA documentation.
Can a single bad fiber in an MPO trunk cause NCCL bandwidth to drop?
Yes. A 12-fiber MPO trunk carries 12 lanes of InfiniBand (e.g., 12x 400 Gb/s for NDR). If one fiber is bad, that lane drops to a lower speed or goes down, reducing aggregate bandwidth by at least 1/12 (8.3%) plus retransmission overhead from the remaining lanes. The link may still show as up if the switch auto-negotiates to a lower speed. Use ibstat to check per-lane speed and errors, then inspect and clean the connector.