Networking_
InfiniBand NDR/XDR vs RoCE: What Changes for the Cable Plant
Details how choosing InfiniBand NDR/XDR versus RoCE for the scale-out fabric changes MPO trunk selection, patching sequences, cleaning protocols, and test parameters during rack deployment and inter-rack cabling for GPU clusters.
Key facts
- NVL72-class racks keep all GPU-to-GPU NVLink traffic on internal copper backplanes; fiber and MPO carry only the scale-out InfiniBand or Ethernet fabric between switches and racks.
- MPO trunk cables for 400G and 800G fabrics are factory-terminated and polished; field crews perform only routing, cleaning, inspection, and patching.
- InfiniBand NDR uses 4x 100 Gb/s lanes per port while XDR doubles that to 4x 200 Gb/s; RoCE implementations on Ethernet follow the same lane counts but under IEEE 802.3 specifications.
- TIA-568.3-D and IEC 61754-7 define MPO connector geometry and polarity; crews must verify Type A, B, or C polarity before installation to avoid crossed lanes.
- IEC 61300-3-35 sets the inspection criteria for end-face cleanliness; a single scratch or particle above the allowed zone fails the link at 400G and above.
- An OTDR or calibrated MPO continuity tester is required after every patch to confirm insertion loss and polarity before the fabric is brought online.
- Leviathan Systems crews route MPO trunks in separate pathways from power and liquid-cooling lines to maintain minimum bend radius and avoid EMI coupling.
Scale-out fabric domain versus intra-rack NVLink
In NVL72-class racks the NVLink domain stays on copper backplanes inside the rack enclosure. All fiber and MPO work therefore belongs exclusively to the scale-out network that connects switches to switches and racks to racks.
This separation means cable-plant crews never use MPO trunks to carry NVLink traffic. The choice between InfiniBand NDR/XDR and RoCE only affects the optics, lane count, and switch port types on the fabric side.
Deployment teams therefore plan two distinct cable schedules: one for internal copper assemblies and one for the external MPO trunks that carry either InfiniBand or Ethernet frames.
MPO trunk and transceiver differences by fabric type
InfiniBand NDR ports typically appear as QSFP112 or OSFP modules with four 100 Gb/s lanes. XDR doubles the per-lane rate to 200 Gb/s, requiring tighter loss budgets and often parallel single-mode or multimode trunks with higher fiber counts.
RoCE deployments on Ethernet use the same physical QSFP/OSFP form factors but must comply with IEEE 802.3 lane specifications and forward-error-correction settings. The MPO connector itself remains identical; only the transceiver firmware and switch configuration change.
Because trunks are factory-terminated, crews select the correct fiber type and polarity at ordering time rather than attempting field termination. Mismatched fiber grade or polarity forces a re-pull after testing.
Routing, slack management, and pathway separation
MPO trunks must maintain the OEM-specified minimum bend radius throughout the rack and row. Crews install trunks after the copper NVLink assemblies but before liquid-cooling manifolds are pressurized, reducing the chance of accidental kinking.
Separate pathways for fiber trunks, power whips, and coolant lines prevent both mechanical damage and EMI coupling. Slack loops are stored in designated vertical managers so that future moves-adds-changes do not disturb live links.
Leviathan Systems documents each trunk route with photographs and length measurements so that replacement orders match the original geometry exactly.
Cleaning, inspection, and polarity verification sequence
Every MPO connector is cleaned with the OEM-approved cassette or pen before insertion. An end-face microscope check against IEC 61300-3-35 follows immediately; any defect aborts the step and requires re-cleaning.
Polarity is confirmed with a dedicated MPO tester before the far end is patched. Type B trunks are the most common for parallel optics, but crews verify the actual mapping rather than assuming the label is correct.
Only after both ends pass inspection and polarity does the crew record the link in the project database and move to the next trunk.
Test parameters and acceptance criteria
Insertion loss is measured with a calibrated MPO tester or OTDR in both directions. The acceptable loss value is taken directly from the transceiver datasheet for the chosen speed and reach; no generic dB allowance is used.
Return loss and lane-by-lane continuity are also recorded. Any lane that fails continuity indicates a polarity error or damaged ferrule that must be corrected before the switch port is enabled.
Results are uploaded the same day so that the commissioning team can proceed with switch bring-up only on validated links.
Common field failure modes and detection points
The most frequent MPO failure is contamination introduced after cleaning but before mating. A single particle on the ferrule end-face produces high loss at 400G and 800G; crews catch it only by inspecting immediately before insertion.
Polarity errors appear when trunks are pulled from the wrong reel or when patch panels are mislabeled. These are detected at the continuity-test step; discovering them after switches are cabled wastes hours of rework.
Bend-radius violations during routing cause intermittent high-bit-error rates that surface only under full fabric load. Documented pathway photographs and post-installation OTDR traces allow crews to locate and correct the kink before production traffic begins.
Standards referenced: TIA-568.3-D · IEC 61754-7 · IEC 61300-3-35 · IEEE 802.3
Frequently asked_
Do we need different MPO trunk part numbers for InfiniBand NDR versus RoCE at the same speed?
The physical MPO trunk is the same; selection is driven by fiber type, count, and polarity rather than protocol. Transceiver and switch configuration determine whether the link runs InfiniBand or Ethernet. Crews still verify the loss budget against the specific optic datasheet before ordering.
When should MPO inspection occur relative to switch installation?
Inspection and cleaning happen immediately before each connector is mated to its patch panel or transceiver. Performing the check earlier allows dust to settle again. Switch installation follows only after all trunks on that panel have passed both inspection and continuity testing.
What tool replaces a standard power meter when validating 400G and 800G parallel links?
A calibrated MPO continuity tester or OTDR that supports the fiber count and polarity scheme is required. It reports per-lane loss and confirms mapping in one operation. Generic duplex power meters cannot verify all lanes or polarity at these speeds.
How does Leviathan Systems handle trunk replacement after a polarity failure is found?
The failed trunk is left in place until a matching replacement arrives. The new trunk is pre-tested on the bench for polarity and loss, then swapped end-to-end while the rest of the row remains live. Both ends are re-inspected and re-tested before the link is returned to the fabric.
Can field crews crimp or re-terminate MPO connectors on site?
No. MPO trunks for NDR and XDR fabrics are factory-terminated and polished. Field work is limited to routing, cleaning, inspection, patching, and testing. Any damaged ferrule requires a complete trunk replacement.