Networking_
Validating an InfiniBand Fabric with ibdiagnet: Errors, Width, and Routing
This article details the exact ibdiagnet command sequence and output checks required to confirm every InfiniBand scale-out link in an NVL72-class deployment trains at full width and speed before cluster handover.
Key facts
- ibdiagnet -o /tmp/ibdiagnet2 runs a full fabric discovery and writes per-port counters plus topology files to the output directory.
- Link width appears in the ibdiagnet2.log as Active Width with values of 1x/2x/4x/8x/12x; any value below the expected port width indicates a training failure.
- The tool reports symbol errors, CRC errors, and link-down events from the PortCounters MADs; thresholds are set in the IBTA specification for acceptable BER.
- Routing validation uses the extracted LFT tables to confirm every destination LID has a valid next-hop entry without dead ends.
- MPO trunk cables carrying the scale-out fabric must be inspected with a calibrated MPO continuity tester and cleaned before ibdiagnet runs; dirty ferrules are the most common cause of width negotiation failures.
- Leviathan Systems includes ibdiagnet output review in the standard commissioning checklist for every rack row handed over to the operator.
- An OTDR trace on suspect fiber runs is required when ibdiagnet shows intermittent CRC errors that do not correlate with port width.
Running the initial fabric discovery
Execute ibdiagnet -o /tmp/ibdiagnet2 -c from a management node that has full subnet manager visibility. The -c flag forces a fresh cable-info query so that every link state is captured before any counter wrap occurs.
Review the generated ibdiagnet2.log first for the total number of nodes and switches discovered. Compare this count against the as-built rack elevation list; any missing GUID indicates an unplugged or failed HCA or switch port that must be resolved before width checks proceed.
Confirming link width and speed on every port
Open the ports.txt file produced by ibdiagnet and filter for Active Width and Active Speed columns. Every port must show the full width and speed defined by its type in the OEM specification; any reduced entry means the link trained down and will not deliver the designed bisection bandwidth.
Cross-reference the local and remote port GUIDs against the cable schedule. A width mismatch on only one end of a trunk usually points to a bent MPO latch or an incompletely seated connector rather than a cable defect.
Reviewing error counters and BER
Examine the PortCounters section for symbol errors and CRC errors accumulated during the scan window. Acceptable counts remain below the IBTA-defined threshold for the link speed; any port exceeding that threshold requires isolation and re-testing after cleaning.
Run a second pass with ibdiagnet -o /tmp/ibdiagnet2 -pc to reset counters and capture a clean sample. This delta confirms whether errors are accumulating under load or were only historical.
Validating routing tables and LFT consistency
Use the LFT files written by ibdiagnet to verify every destination LID has a programmed next-hop port on each switch. Missing or duplicate entries produce silent black-hole routes that only appear under all-to-all traffic.
Compare the extracted topology against the intended fat-tree or dragonfly layout. Any switch whose LFT shows routes that bypass an expected uplink indicates a subnet manager configuration error that must be corrected before production traffic.
Common failure modes observed in the field
The most frequent width negotiation failure occurs when an MPO connector is inserted with one latch only partially engaged; the link trains at reduced width and ibdiagnet reports the value immediately. Always re-seat the connector and re-run the scan rather than assuming a cable fault.
Dirty end-faces produce elevated CRC counts without width reduction. An MPO inspection scope must be used on every trunk before the first ibdiagnet pass; Leviathan Systems mandates this step on every row to prevent later packet-loss surprises during MPI benchmarks.
Power-cycle timing on switches can leave stale LFT entries. After any switch reboot, wait for the subnet manager to complete a full sweep before trusting ibdiagnet routing output.
Documenting results for cluster handover
Archive the entire /tmp/ibdiagnet2 directory with a timestamp and rack-row identifier. The operator receives both the raw logs and a one-page summary listing any ports that required remediation.
Include the final ibdiagnet pass in the commissioning package so that subsequent baseline comparisons can detect degradation over time. This record also serves as evidence that every scale-out link met the full-width requirement before the cluster was accepted.
Standards referenced: IBTA InfiniBand Architecture Specification Volume 1 (link training and PortCounters MADs) · IBTA InfiniBand Architecture Specification Volume 2 (subnet management and LFT programming)
Frequently asked_
How long does a full ibdiagnet run take on a 1000-node fabric?
A complete discovery plus counter read finishes quickly on fabrics of this size when the subnet manager is already stable. Larger fabrics with many switches extend the time proportionally to the number of PortInfo queries.
Does ibdiagnet require the fabric to be idle?
No production traffic is required, but heavy all-to-all patterns can increment counters during the scan window. Run the tool during a maintenance window or after resetting counters so that any reported errors reflect link integrity rather than workload.
What action follows when ibdiagnet shows reduced width on a port?
Re-seat both MPO connectors on that trunk, clean the ferrules, and repeat the scan. If width remains reduced, replace the trunk cable; the fault is almost always mechanical seating or contamination rather than a switch or HCA silicon issue.
Can ibdiagnet be scripted into the rack commissioning workflow?
Yes. Leviathan Systems includes a wrapper that launches ibdiagnet after each row of racks is cabled, parses the width column, and fails the job if any port is below target width. The script also emails the summary to the deployment lead before the next row begins.
Is an OTDR needed on every MPO trunk?
Only when CRC errors appear without a width reduction. A calibrated MPO continuity tester plus end-face inspection catches the majority of issues; OTDR is reserved for trunks that pass visual inspection yet still show bit errors.