What Is NCCL Bandwidth Test?_
The NCCL bandwidth test (commonly run as `nccl-tests`) evaluates the throughput of GPU-to-GPU communication within a node or across nodes via NVLink or network fabrics. It reports unidirectional and bidirectional bandwidth for operations like all-reduce or all-gather, helping to validate that interconnects are performing to spec. The test is a standard step during commissioning to catch cabling faults, driver misconfigurations, or thermal throttling before production workloads are deployed.
Technical Details
The test uses NCCL’s internal algorithms to saturate copper NVLink bridges within a node and separate scale-out InfiniBand or Ethernet links, reporting results in GB/s. It runs multiple iterations with varying message sizes (e.g., 256 bytes to 512 MB) to expose latency and bandwidth bottlenecks. Results are compared against expected baselines from the OEM specification for the given GPU model and interconnect topology (e.g., H100 NVLink ring). The test does not exercise liquid cooling or power delivery directly, but degraded performance can indicate thermal or electrical issues that affect GPU clock speeds.
How Leviathan Systems Works with NCCL Bandwidth Test
Our field crew runs NCCL bandwidth tests after rack assembly, structured cabling, and liquid cooling hookup to confirm that all GPU-to-GPU links meet the required throughput before handing off to the customer. A failing test often points to a loose NVLink bridge, a mis-seated OSFP or QSFP cable, or an incorrectly configured switch partition.
Related service
Network Testing & Commissioning →