Networking_
Rail-Optimized vs Fat-Tree: The Field Wiring Plan, Port by Port
A field engineer's definitive guide to physically wiring and patching rail-optimized versus fat-tree InfiniBand/Ethernet fabrics in AI data centers, port by port, including common failure modes and testing procedures.
Key facts
- Rail-optimized topologies require each GPU node's NIC to connect to a single leaf switch port, with all NICs in the same GPU 'rail' homed to the same leaf, enabling one-hop GPU-to-GPU communication across the fabric.
- Fat-tree topologies use a Clos architecture where each leaf switch connects to multiple spine switches, requiring equal-cost multipath (ECMP) hashing and typically 2:1 oversubscription at the spine layer.
- In an NVL72 rack, the 72 GPUs are grouped into 9 rails of 8 GPUs each; each rail's 8 NICs must be patched to the same leaf switch (or same set of leaf ports) to maintain rail locality.
- MPO trunk cables used for scale-out fabrics are factory-terminated with APC or UPC polish; field work is limited to patching, routing, cleaning, and testing—never field-crimping MPO ferrules.
- A single dirty or damaged MPO connector can cause link errors at the physical layer, which manifest as CRC errors or flapping links, often misdiagnosed as a higher-layer issue.
- The relevant standard for MPO connector inspection is IEC 61300-3-35, which defines pass/fail criteria for scratches, pits, and contamination on the endface.
- OTDR testing of each MPO trunk cable after installation verifies end-to-end loss and identifies any bends, breaks, or bad connectors before the fabric is commissioned.
Rail-Optimized vs Fat-Tree: The Physical Wiring Map
In a rail-optimized topology, each GPU node's NIC is assigned to a specific 'rail'—a group of GPUs that share a common leaf switch. For example, in an NVL72 rack with 72 GPUs, the GPUs are divided into 9 rails of 8 GPUs each. All 8 NICs from the same rail must be patched to the same leaf switch (or to a dedicated set of leaf ports on a modular switch). This design ensures that any GPU can reach any other GPU within the same rail in one hop through the leaf switch, and cross-rail traffic goes up to the spine layer. The physical wiring plan must match the logical rail assignment: each leaf switch's ports are pre-assigned to a specific rail, and the trunk cables from the rack's top-of-rack (ToR) patch panel to the leaf switch must be labeled accordingly.
In a fat-tree topology, the wiring is more uniform. Each leaf switch connects to every spine switch in a full-mesh pattern, typically using equal-cost multipath (ECMP) for load balancing. The physical cabling is a Clos network: for a 2-tier fat-tree, each leaf switch has a number of uplinks to the spine layer, and each spine switch connects to all leaf switches. The oversubscription ratio is commonly 2:1, meaning the aggregate uplink bandwidth from a leaf to the spine is half the aggregate downlink bandwidth to the GPUs. The wiring plan must ensure that each leaf switch has the same number of uplinks to each spine switch to maintain symmetric routing. Unlike rail-optimized, there is no GPU-to-leaf affinity; any GPU can reach any other GPU through any leaf-spine path, but ECMP hashing may cause inconsistent latency.
Port-by-Port Patch Schedule for Rail-Optimized Fabrics
The patch schedule for a rail-optimized fabric must be meticulously documented and followed. Start by labeling each GPU node's NIC port with its rail number and node index. For an NVL72 rack, rails are numbered 0 through 8, and nodes within a rail are 0 through 7. The leaf switches are also labeled by rail number. The patch panel in the rack's ToR position should have ports grouped by rail. The trunk cables from the rack to the leaf switches must be routed in bundles per rail, with each bundle containing exactly 8 fibers (one per NIC). Use color-coded MPO trunk cables or cable ties to differentiate rails. The patch schedule should list, for each rail, the leaf switch port range (e.g., ports 1-8 on leaf switch 0) and the corresponding NIC ports on the GPU nodes.
When patching, always clean and inspect each MPO connector before insertion. Use a one-click cleaner for the endface and an inspection scope with a pass/fail algorithm per IEC 61300-3-35. After patching, verify link status at the leaf switch CLI: check for 'link up' on all expected ports. Then run a connectivity test from a GPU node to the leaf switch using a simple ping or a fabric management tool like 'ibdiagnet' for InfiniBand. Document any link failures and re-inspect the connector. A common mistake is to assume a link failure is a bad cable; often it is a dirty connector or a bent fiber at the patch panel.
Port-by-Port Patch Schedule for Fat-Tree Fabrics
For a fat-tree fabric, the patch schedule is driven by the leaf-to-spine uplink count and the spine switch count. Suppose you have 8 leaf switches and 4 spine switches. Each leaf switch needs 4 uplinks to the spine layer, one to each spine switch. The patch schedule must specify which leaf switch ports connect to which spine switch ports. Typically, the first 4 ports on each leaf switch are reserved for uplinks, and the remaining ports are for GPU NICs. The trunk cables from the leaf switches to the spine switches should be routed in a structured cable management system, with each trunk labeled with the source leaf and destination spine. Use a standard labeling scheme: e.g., 'L01-S01' for leaf 1 to spine 1.
When patching the GPU NICs to the leaf switches, there is no rail affinity, so any leaf switch port can connect to any GPU NIC. However, to simplify troubleshooting, assign each leaf switch to a specific rack or group of racks. For example, leaf switches 0-3 serve rack A, and leaf switches 4-7 serve rack B. The patch schedule should list each GPU NIC's MAC address or GUID, the leaf switch port, and the rack location. After patching, verify that all leaf switch ports show 'link up' and that the spine switches see all leaf switches as neighbors. Run a fabric-wide connectivity test using a tool like 'ibnetdiscover' for InfiniBand or 'lldpctl' for Ethernet to confirm the full mesh is intact.
Common Failure Modes in the Field
The most frequent failure mode is a dirty or damaged MPO connector endface. Dust, oil, or scratches cause back reflection and insertion loss, leading to CRC errors, flapping links, or complete link failure. This is often misdiagnosed as a bad transceiver or cable. Always inspect every connector before insertion, and re-inspect if a link fails to come up. Another common issue is incorrect polarity: MPO trunk cables can be Type A, B, or C, and mismatched polarity between the patch panel and the switch transceiver will cause a link failure. Verify polarity per the TIA-568.5 standard and use a polarity tester before commissioning. Polarity errors can be caught early with a simple continuity check using a visual fault locator or a dedicated tester.
Bend radius violations are another failure mode. Fiber optic cables, especially MPO trunk cables, have a minimum bend radius specified by the manufacturer; exceeding this causes micro-bends that increase loss and may damage the fiber. This often happens when cables are routed through tight spaces in the rack or under floor tiles. Use cable management arms and vertical cable managers to maintain the proper radius. Loose or mismatched transceivers can also cause intermittent links. Ensure the transceiver type (e.g., 100G SR4 vs 100G PSM4) matches the fiber type (multimode vs single-mode) and that the transceiver is fully seated in the switch port. Finally, mis-patching—connecting a NIC to the wrong leaf switch or the wrong port—is a human error that can be prevented by a double-check with a printed patch schedule and a second engineer verifying each connection.
Testing and Commissioning Procedures
After all cabling is installed and patched, perform a three-phase test. Phase 1: Visual inspection and cleaning of every MPO connector endface using an inspection scope with automated pass/fail per IEC 61300-3-35. Phase 2: Optical loss testing using an OTDR or light source and power meter. For each trunk cable, measure end-to-end loss and compare to the calculated link budget based on the connector count (typically 0.5 dB per pair worst-case) and fiber attenuation (per the fiber datasheet). If loss exceeds the budget, locate the fault with the OTDR and re-terminate or replace the cable. Phase 3: Link-level testing at the switch CLI. Verify all ports show 'link up' and no CRC errors or link flaps over a 10-minute soak test.
For InfiniBand fabrics, run 'ibdiagnet -r' to verify the fabric topology and check for miswirings. For Ethernet, use 'lldpctl' to confirm neighbor relationships and 'ping' across the fabric. Document all test results, including OTDR traces and link error counts. If any link fails, isolate the issue: swap the transceiver, re-inspect the connector, or replace the trunk cable. Leviathan Systems follows this exact procedure on every deployment to ensure zero link errors at handoff. A final step is to label each trunk cable with a unique ID that matches the patch schedule, using a thermal transfer printer with self-laminating labels per TIA-606-B.
Tools and Standards for Field Work
Essential tools include a fiber inspection scope with 200x or 400x magnification and automated pass/fail software, a one-click MPO cleaner, an OTDR with MPO launch cables, and a polarity tester. For cleaning, use only lint-free wipes with isopropyl alcohol (99% purity) or a dry-cleaning system designed for MPO connectors. Never use compressed air—it can deposit oil and cause contamination. The relevant standards are IEC 61300-3-35 for connector inspection, TIA-568.5 for MPO polarity types, and TIA-606-B for labeling administration. For bend radius, follow the manufacturer's specification; a general rule is 10× the cable diameter during installation and 15× for permanent routing, but always verify with the cable datasheet.
For labeling, use a thermal transfer printer with self-laminating labels to ensure durability. Label both ends of every trunk cable with a unique ID that matches the patch schedule. Use a consistent naming convention, such as 'RACK-A-LEAF-01-PORT-01' to 'RACK-A-GPU-RAIL-0-NODE-0'. This makes troubleshooting faster and reduces human error. Leviathan Systems maintains a digital copy of the patch schedule and test results for every deployment, which is handed over to the customer for ongoing operations.
Standards referenced: IEC 61300-3-35 · TIA-568.5 · TIA-606-B
Frequently asked_
How do I know if my fabric is rail-optimized or fat-tree from the wiring plan?
Look at the leaf switch port assignments. In a rail-optimized fabric, each leaf switch's ports are dedicated to a specific group of GPUs (a rail), and the uplinks to the spine are typically fewer (e.g., 2 uplinks per leaf). In a fat-tree, every leaf switch has the same number of uplinks to every spine switch, and the GPU NICs are distributed evenly across all leaf switches without any rail affinity. The patch schedule will explicitly state the topology type.
What is the most common cause of link failures after patching?
Dirty or damaged MPO connector endfaces. Even a single speck of dust can cause enough back reflection to prevent a link from coming up. Always inspect and clean every connector before insertion using an endface scope with automated pass/fail per IEC 61300-3-35. The second most common cause is polarity mismatch—using a Type A cable when a Type B is needed, or vice versa. Use a polarity tester to verify before commissioning.
Can I use the same MPO trunk cables for both rail-optimized and fat-tree fabrics?
Yes, the same MPO trunk cables (e.g., 12-fiber or 24-fiber) work for both topologies. The difference is in the patch schedule and how the cables are routed. In a rail-optimized fabric, you bundle cables per rail; in a fat-tree, you bundle cables per leaf-to-spine mesh. The cable type (single-mode or multimode) depends on the transceiver and distance, not the topology. Ensure the fiber type matches the transceiver specification (e.g., SR4 for multimode, PSM4 for single-mode).
How do I test polarity in the field without expensive equipment?
Use a simple handheld MPO polarity tester that checks continuity and polarity type (A, B, or C). Alternatively, you can use a visual fault locator (VFL) with a red laser to trace fibers and check continuity, but this does not confirm polarity type. For a full fabric, an OTDR with MPO launch cables is recommended to measure loss and locate faults; some OTDRs also have polarity-checking features.
What is the typical timeline for wiring a 72-GPU rack in a rail-optimized fabric?
For a single rack, expect 4-6 hours for cable routing, patching, and labeling, assuming the trunk cables are pre-terminated and the patch panels are already installed. Add 2-3 hours for testing and commissioning (inspection, OTDR, link tests). For a large deployment of 100+ racks, Leviathan Systems typically completes wiring and testing in 2-3 weeks with a crew of 8-10 engineers, depending on site conditions and cabling complexity.