Liquid Cooling_
Air-to-Liquid Cooling Retrofit: The Install Side
A step-by-step field guide for converting an air-cooled GPU data center hall to liquid cooling, covering the physical retrofit sequence, infrastructure prerequisites, and common pitfalls, written for deployment engineers and operators.
Key facts
- Liquid cooling retrofits require overhead clearance per the CDU and piping OEM specs (typically 2.4 m or more, per ASHRAE TC 9.9 guidelines).
- The retrofit sequence must install the facility-side coolant loop and CDUs before any rack-level work to avoid cross-contamination of the GPU cold plates.
- Dry-break quick disconnects are mandatory on all rack-level coolant lines to allow hot-swap of servers without draining the entire loop.
- Pressure testing of the facility loop must hold at 1.5× the operating pressure for 24 hours with zero measurable drop, per ASME B31.9.
- Air-cooled GPU racks typically draw 15–30 kW; liquid cooling retrofit can support 60–120 kW per rack, requiring upstream electrical upgrades.
- Coolant purity must meet OEM resistivity and particle count specs (e.g., >1 MΩ·cm and <100 particles/mL >0.5 μm) to prevent galvanic corrosion in cold plates.
- The transition from air to liquid cooling reduces fan power by 60–80% but adds pump power of 2–5 kW per CDU, netting a 30–50% reduction in total cooling energy.
Pre-Retrofit Infrastructure Assessment and Hall Preparation
Before any piping work begins, verify that the hall can physically support liquid cooling. The most common showstopper is overhead clearance: CDUs and their associated piping require clear vertical space per the OEM specs—typically 2.4 m or more, per ASHRAE TC 9.9 guidelines. If your hall has cable trays, lighting, or fire suppression drops below that, relocate them. Also check floor loading: a fully loaded CDU with coolant can exceed 1,500 kg, so confirm the slab rating meets the CDU manufacturer's requirements.
Next, confirm that the facility water supply (building chilled water or a dedicated coolant loop) can deliver the required flow rate and temperature. For a hall with 100 racks at 80 kW each, you need roughly 800 L/min of coolant at a supply temperature of 18–22°C, but these numbers vary by OEM design. If your existing chilled water system was designed for air handlers, it may lack the pressure head or flow capacity. You will likely need a secondary pump skid and a heat exchanger to isolate the facility loop from the rack loop. Finally, plan CDU placement: they should be within 15 m of the racks to minimize pressure drop in the distribution hoses, and positioned to allow service access on all sides.
Facility-Side Coolant Loop Installation and Pressure Testing
The facility-side loop is the backbone of the retrofit. Use schedule 40 or 80 stainless steel or copper piping, sized per the total heat load. For a 5 MW hall, a 4-inch main header is typical. All joints must be welded or flanged—no compression fittings in the main loop. Install isolation valves at every branch to each CDU so you can service one without draining the entire hall. The loop must be flushed with deionized water and filtered to 50 µm before filling with the final coolant.
Pressure testing is non-negotiable. Fill the loop with deionized water, pressurize to 1.5× the operating pressure (per ASME B31.9), and hold for 24 hours. Any pressure drop indicates a leak—use a calibrated digital pressure gauge with 0.1 psi resolution. After the test, drain and refill with the final coolant mixture (typically 30% propylene glycol and 70% deionized water, with corrosion inhibitor per OEM spec). Verify coolant resistivity is above 1 MΩ·cm using a handheld conductivity meter; if it's lower, the coolant will cause galvanic corrosion in GPU cold plates within weeks.
Rack-Level Conversion: Removing Air Cooling and Installing Liquid Cooling Hardware
Once the facility loop is live and tested, work on one rack at a time. Power down the GPUs and disconnect power cables. Remove the air-cooled heat sinks and fans—these are typically held by captive screws. Clean the GPU die and baseplate with isopropyl alcohol and lint-free wipes. Apply thermal interface material (TIM) per the OEM spec: a uniform layer of phase-change pad or thermal grease. Install the cold plate, torque the mounting screws to the OEM specification (typically in a cross pattern), and connect the coolant hoses using dry-break quick disconnects.
Route the hoses to the rack manifold, which distributes coolant from the CDU to each server. Use a hose management tray to keep bends above the minimum bend radius—typically 5× the hose outer diameter. Label every hose with rack and server ID. After all servers in a rack are converted, connect the rack manifold to the CDU via the facility loop. Power on the CDU and verify flow rate per server: typically 1–2 L/min per GPU, depending on the model. Check for leaks at every connection with a paper towel—any moisture means a loose fitting or damaged O-ring.
Commissioning the Cooling Loop and Verifying Thermal Performance
Commissioning starts with the CDU. Set the supply temperature to the OEM-recommended value (typically 20–25°C) and the pump speed to 50%. Monitor the return temperature; it should rise by 5–8°C at full load. Use the CDU's flow meters to confirm each rack manifold is receiving the design flow. If a rack is under-flowing, check for kinked hoses or a partially closed isolation valve. Next, power on the GPUs and run a stress test that loads all cores to 100%. Monitor GPU temperatures via nvidia-smi; they should stabilize below 85°C for air-cooled reference, but with liquid cooling they typically run 50–65°C at full load.
If temperatures are higher than expected, check the TIM application and cold plate contact. Common issues include uneven torque or a warped baseplate. Also verify the coolant flow rate per GPU using the CDU's per-port flow meters. If flow is low, the dry-break connector may not be fully seated. Finally, log the data for 24 hours to confirm stability. Any temperature excursion above the OEM limit (often 90°C) warrants immediate shutdown and inspection.
Common Failure Modes in Air-to-Liquid Retrofit and How to Catch Them
The most frequent failure is coolant contamination. If the facility loop was not properly flushed, debris can clog the cold plate microchannels, causing a GPU to overheat within hours. Catch this by installing a 50 µm filter at the CDU inlet and checking it after the first 24 hours of operation. Another common issue is galvanic corrosion from using dissimilar metals—for example, aluminum fittings on copper cold plates. Always use brass or stainless steel fittings and verify coolant resistivity weekly.
Leaks at dry-break connectors are the second most common failure. They occur when the connector is not fully seated or the O-ring is damaged during installation. To catch this, perform a visual inspection of every connection after the initial fill, then pressurize the rack loop to 1.2× operating pressure and check with a soap solution—any bubbles indicate a leak. Finally, pump cavitation can occur if the CDU is installed above the rack level, causing air to enter the pump. Always install CDUs at or below the lowest rack to maintain positive suction head. If cavitation happens, you'll hear a rattling sound and see fluctuating flow readings; vent the pump immediately.
Transitioning the Hall from Air to Liquid: Phasing and Cutover Strategy
Do not attempt to convert the entire hall in one weekend. Phase the retrofit by row or by zone. Start with one row of racks—typically the highest-density row—and convert those to liquid cooling while the rest of the hall remains air-cooled. This requires temporary barriers (plastic sheeting or insulated panels) to prevent hot air from the air-cooled rows from mixing with the liquid-cooled row's cold aisle. The CDU for that row must be installed and commissioned before any rack conversion begins.
During cutover, plan for a 4-hour window per rack: 1 hour to power down and disconnect, 1 hour to install cold plates and hoses, 1 hour to connect to the manifold and leak test, and 1 hour to power up and verify. Have a rollback plan: if a cold plate leaks, you must be able to reinstall the air-cooled heat sink and fans within 30 minutes. Keep a spare set of air-cooled hardware on a cart for each rack. After the first row is stable for 72 hours, proceed to the next row. This phased approach minimizes risk and allows you to train your crew on the process before scaling.
Electrical and Network Considerations for Higher Power Density
Liquid cooling enables higher power density per rack, but that means your electrical distribution must be upgraded. Typical air-cooled GPU racks draw 15–30 kW; after liquid cooling, the same rack can draw 60–120 kW. Verify that your PDUs, busways, and breakers are rated for the new load. You may need to install a second power feed per rack or upgrade to 415 V three-phase. Also check the UPS capacity: a 5 MW hall will need a 5 MW UPS, not the 2 MW unit you had for air cooling.
On the network side, the higher density means more GPUs per rack, which increases the number of NVLink (copper) and InfiniBand/Ethernet (fiber) cables. Ensure your cable management trays have enough capacity—typically 50% more than the air-cooled design. Use pre-terminated MPO trunks for fiber to avoid field termination. Label every cable at both ends and in the tray. The increased power density also generates more heat in the network switches; ensure they are either in the liquid-cooled loop or have dedicated air cooling if they are not converted.
Standards referenced: ASHRAE TC 9.9 · ASME B31.9 · ISO 4406 for coolant cleanliness · IEC 60529 for dry-break connector ingress protection
Frequently asked_
Can I retrofit a rack that is still running air-cooled GPUs without shutting down the entire hall?
Yes, but only if you phase the work by row and use temporary barriers to isolate the liquid-cooled row from the air-cooled rows. You must also have a separate CDU for each row. The GPUs in the row being converted must be powered down during the hardware swap, but the rest of the hall can remain operational. Leviathan Systems typically does this during a planned maintenance window, converting one row per night to minimize downtime.
What coolant should I use, and how often do I need to replace it?
Use a mixture of 30% propylene glycol and 70% deionized water with a corrosion inhibitor, per the OEM spec. Replace the coolant every 3–5 years or when resistivity drops below the OEM threshold (typically 1 MΩ·cm). Test the coolant quarterly with a handheld conductivity meter and a particle counter. If you see particles above 100 per mL at 0.5 µm, flush the loop and replace the coolant.
How do I handle leaks in the dry-break connectors during operation?
If a dry-break connector leaks, isolate the server by closing the rack manifold valve for that port. Disconnect the hose, inspect the O-ring for damage, and replace it if needed. Reconnect and verify the seal with a soap solution. If the connector body is cracked, replace the entire connector. Always keep spare O-rings and connectors on hand. Leviathan Systems stocks a full kit for each rack type.
What is the typical payback period for converting an air-cooled hall to liquid cooling?
The payback period depends on your electricity cost and utilization. For a 5 MW hall running at 80% load, the reduction in fan power and improved cooling efficiency typically yields a 30–50% reduction in cooling energy costs. At $0.10 per kWh, that saves roughly $130,000 per year. With a retrofit cost of $500,000 to $1 million, payback is 4–8 years. However, the primary benefit is enabling higher density and performance, not just energy savings.
Do I need to upgrade my fire suppression system for a liquid-cooled hall?
Yes, because liquid cooling introduces a large volume of flammable coolant (propylene glycol) and electrical equipment in close proximity. You need a fire suppression system rated for Class A, B, and C fires. This typically means a clean agent system (e.g., FM-200 or Novec 1230) with pre-action sprinklers. Also install leak detection sensors under every CDU and rack manifold, connected to the BMS for automatic shutdown.