Field Notes_
Data Center Migration for AI Infrastructure: A Practical Field Guide
A practical field guide for relocating GPU infrastructure in AI data centers, detailing the physical-layer sequence to minimize downtime and prevent damage during migration.
Key facts
- GPU rack relocation requires a strict power-down sequence: de-energize liquid cooling loops before rack power to prevent thermal shock.
- MPO trunk cables for scale-out networks (InfiniBand or Ethernet) must be disconnected and labeled per TIA-606-B standards before rack movement.
- Copper NVLink spine cables inside NVL72 racks are factory-terminated and must not be field-crimped; only patch, route, and test per OEM specs.
- Rack leveling and alignment to a tolerance of ±2 mm per TIA-942 is critical to prevent fiber strain and cooling loop leaks.
- Post-migration OTDR testing of all MPO links must show loss within 0.75 dB per connector pair per IEC 61753-1 for 50/125 µm multimode fiber.
- Liquid cooling loop reconnection requires pressure testing to the OEM-specified threshold (e.g., 1.5× operating pressure) before power-up.
- Commissioning after migration includes nvidia-smi NVLink status verification (copper spine only) and network fabric validation via ibdiagnet or equivalent.
Pre-Migration Planning and Risk Assessment
Before touching a single rack, document the current physical-layer topology: every GPU rack’s power feed, liquid cooling loop, and MPO trunk cable must be tagged per TIA-606-B. Use a calibrated MPO continuity tester to verify existing link health—this baseline catches latent damage that could be blamed on the move. Identify the power-down sequence: for liquid-cooled racks, shut down the coolant distribution unit (CDU) first, then the rack PDU, then the GPU servers. This prevents thermal shock from cold coolant hitting hot GPUs. Coordinate with the facility team to ensure the target location has matching power (voltage, phase, amperage) and cooling capacity—mismatched infrastructure is the top cause of migration delays.
Physical Disconnection and Cable Management
Disconnect all MPO trunk cables from the rack’s top-of-rack switches, not from the spine switches—this limits the number of re-terminations needed. Use a fiber cleaning kit (dry clicker or wet-dry cassette) on every MPO connector before capping it with a dust cap. For copper NVLink spine cables inside the rack, do not unplug them unless the rack is being disassembled; they are factory-terminated and field reconnection risks damage. Label every cable with its source and destination per the pre-migration map. For liquid cooling loops, drain the rack’s manifold per the OEM procedure, then cap the quick-disconnects with clean plugs—never reuse caps that have been on the floor.
Rack Movement and Re-Leveling
Move the rack using a pallet jack or rack lifter rated for the total weight (including GPUs and coolant weight). Secure all internal components with foam blocks or straps to prevent GPU sleds from shifting. Once in position, re-level the rack using the leveling feet and a digital level—tolerance per TIA-942 is ±2 mm from horizontal. This is critical because even a 1 mm tilt can misalign liquid cooling quick-disconnects or strain fiber patch cords. Verify the rack is anchored to the floor per seismic or local code, then re-connect facility power and cooling lines with new gaskets or O-rings.
Reconnection and Testing of Liquid Cooling Loops
Reconnect the rack’s coolant supply and return lines to the CDU, using a torque wrench set to the OEM spec for the quick-disconnect fittings. Pressure-test the loop with nitrogen or deionized water at 1.5× the operating pressure—hold for 15 minutes and check for pressure drop. If the drop exceeds the OEM threshold (typically 0.1 bar), locate the leak with a soap solution or ultrasonic detector. Do not power on GPUs until the loop passes pressure test and the coolant temperature is stable within the operating range (e.g., 15–35°C).
Fiber and Copper Network Reconnection
Reconnect MPO trunk cables to the top-of-rack switches, cleaning each connector with a dry clicker before insertion. Use an OTDR to test every link—loss must be within 0.75 dB per connector pair per IEC 61753-1 for multimode fiber. For copper NVLink spine cables, visually inspect the connectors for bent pins or debris, then reseat them firmly. Power on the rack’s network switches first, then the GPU servers. Verify link status with ibdiagnet (InfiniBand) or ethtool (Ethernet) for the scale-out network, and nvidia-smi for NVLink—note that NVLink status reflects the copper spine, not fiber.
Common Failure Modes and How to Catch Them
The most frequent failure is a dirty or damaged MPO connector causing high loss or bit errors—always inspect with a fiber microscope before mating. Another is coolant leaks from O-rings that were not replaced or were cross-threaded—pressure testing catches this before power-up. Third, rack misalignment causing fiber strain: after reconnection, check that patch cords have at least the OEM-specified bend radius (e.g., 10× cable diameter) and no tension. Fourth, power sequencing errors: powering on GPUs before the CDU is stable can cause thermal shock or condensation. Finally, labeling errors: a mislabeled MPO trunk can take hours to trace—use a visual fault locator to confirm each link.
Commissioning and Validation Sequence
After all connections are made, power on the CDU and verify coolant flow rate and temperature are within spec. Then power on the rack PDU and GPU servers one by one, monitoring for overcurrent or thermal events. Run nvidia-smi to confirm all GPUs are detected and NVLink is up (copper spine only). For the scale-out network, run ibdiagnet or a similar fabric checker to verify all links are active and error-free. Finally, run a GPU stress test (e.g., a matrix multiply workload) for 30 minutes to validate thermal and power stability. Document the final OTDR traces and link status for the as-built record.
Standards referenced: TIA-606-B (cable labeling) · TIA-942 (rack leveling tolerance) · IEC 61753-1 (fiber connector loss limits)
Frequently asked_
Do I need to power down the entire data center for a single rack migration?
No, only the rack being moved and its upstream network switches need power-down. Coordinate with the facility team to isolate the rack’s PDU and CDU circuit. Adjacent racks can remain operational if you follow proper cable management and avoid tripping hazards.
Can I reuse MPO trunk cables after a migration?
Yes, but only if they pass OTDR testing after cleaning. MPO connectors are factory-polished and can be reused many times if handled carefully. Always clean and inspect before reconnection—dirt is the primary cause of loss increase.
What is the biggest risk when moving liquid-cooled GPU racks?
Coolant leaks from quick-disconnect fittings that were not properly torqued or had damaged O-rings. Pressure testing before power-up is non-negotiable. Also, thermal shock if the CDU is restarted before the coolant temperature stabilizes.
How do I verify NVLink is working after a rack move?
Run nvidia-smi topo -m to check the NVLink topology. All links should show as 'NV' and the bandwidth should match the expected value (e.g., 400 GB/s per GPU for H100). Note that NVLink runs over the copper spine inside the rack—not over fiber—so fiber issues won't affect it.
Should I hire a specialist crew like Leviathan Systems for a migration?
If your internal team lacks experience with liquid cooling, MPO handling, or GPU rack-leveling, a specialist crew can reduce downtime risk. Leviathan Systems, for example, performs these sequences routinely for hyperscalers and can complete a migration in hours instead of days.