What Is Congestion Control?_
In high-performance GPU fabrics, congestion control detects incipient buffer occupancy at switch egress ports and signals senders to reduce their transmission rate, avoiding packet drops and the resulting retransmission storms. It operates end-to-end between NICs or within the fabric’s switch ASICs, using explicit congestion notification (ECN) or similar marking schemes. Effective congestion control is critical for maintaining low tail latency in collective operations like all-reduce.
Technical Details
For NVLink-based GPU-to-GPU traffic inside the rack, congestion control is typically handled by the NVSwitch fabric’s hardware credit-based flow control, which is lossless by design. On the scale-out InfiniBand or Ethernet fabric, congestion control relies on ECN marking at switches and rate limiters at the host channel adapter (HCA). The relevant standard (e.g., IEEE 802.1Qbb for lossless Ethernet or InfiniBand’s adaptive routing and congestion notification) defines the signaling, but exact thresholds and reaction times are set per OEM spec. Misconfigured congestion control can cause head-of-line blocking that stalls GPU kernel launches.
How Leviathan Systems Works with Congestion Control
During GPU data-center commissioning, our field crew validates that congestion control parameters (e.g., ECN marking thresholds on ToR switches) match the OEM’s reference design for the specific GPU cluster. We often observe that improper tuning leads to performance degradation in distributed training jobs, which we flag for the network team.
Related service
Network Testing & Commissioning →