Liquid Cooling_
Coolant Distribution Unit (CDU) Installation & Commissioning
A field-proven, step-by-step guide to installing and commissioning a Coolant Distribution Unit (CDU) in an AI data center, covering flow balancing, temperature setpoints, redundancy configurations, and the baseline checks that prevent leaks and thermal events.
Key facts
- CDU supply temperature is typically set between 20-25°C for direct-to-chip cooling, with a return delta of 5-10°C depending on GPU load and coolant type — always verify the OEM’s spec for your specific GPU generation.
- Flow rate per GPU cold plate is specified by the OEM (e.g., 1-2 L/min for H100/B200-class GPUs) and must be verified with a calibrated clamp-on ultrasonic flow meter at each branch manifold, not just at the CDU outlet.
- Redundancy configurations (N+1 or 2N) require separate CDUs, each with its own pump, heat exchanger, control loop, and dedicated power feed, sharing a common coolant loop via motorized isolation valves that open on failover.
- Leak detection sensors (both point and cable type) must be placed under every quick-disconnect, valve, and cold plate, wired to a BMS or CDU controller with a response time under 10 seconds; test each sensor during commissioning by applying a known-conductivity fluid.
- The CDU's secondary-side (facility water) inlet temperature must be at least 5°C below the primary-side setpoint to ensure adequate heat rejection across the plate heat exchanger; if this delta cannot be maintained, re-evaluate the facility chiller setpoints.
- Commissioning includes a 24-hour pressure hold test at 1.5x the operating pressure (typically 4-6 bar) with zero pressure drop; use a calibrated pressure transducer and data logger, and if the pressure drops more than 0.1 bar, isolate sections to locate the leak.
- Glycol concentration in the coolant must be verified with a refractometer to the OEM spec (typically 25-40% by volume) to prevent freezing and maintain thermal conductivity; note that too much glycol increases viscosity and reduces heat transfer.
CDU Placement and Mechanical Integration
Position the CDU on a level, vibration-dampened pad within the row or at the row end, ensuring service clearance per the OEM manual — typically 1 meter on the front and rear for filter and pump access. The CDU must be placed to keep the primary-side pressure drop under the pump curve; this usually limits the distance to the farthest rack to 30 meters, but always calculate the actual pressure drop using pipe diameter, flow rate, and number of fittings. Bolt the CDU to the floor using seismic-rated anchors if required by local codes, and connect the facility water supply and return lines to the CDU's secondary-side ports with flexible hoses rated for at least 1.5x the facility loop's maximum pressure. Install isolation ball valves on both secondary-side lines to allow CDU replacement without draining the building loop.
For the primary-side (coolant) loop, run supply and return manifolds from the CDU to each rack using insulated stainless steel or copper pipe sized for the total flow (e.g., 2-inch schedule 40 pipe for up to 200 kW). Install a pressure gauge and a resistance temperature detector (RTD) at each rack's connection point to monitor local conditions. Use a calibrated torque wrench on all flanged connections per the OEM spec (typically 20-30 Nm for 1-inch bolts) and apply thread sealant on NPT fittings. After all mechanical connections are made, perform a visual inspection of every joint and label each line with flow direction, coolant type, and the date of installation.
Coolant Fill, Degassing, and Pressure Testing
Fill the primary loop with the specified coolant — typically deionized water with a corrosion inhibitor and biocide, plus glycol if required — using a fill cart with a 5-micron filter. Start filling from the CDU's reservoir, opening vent valves at the highest points in each rack's manifold to release trapped air. Fill slowly at under 10 L/min to avoid turbulent mixing that can introduce micro-bubbles. Once the loop is full, run the CDU's pump at low speed (20-30% of max) for 30 minutes to circulate and degas; then increase to full speed for another 30 minutes while monitoring the reservoir level and topping off as needed. Repeat the venting process at each rack manifold until no air is heard.
After degassing, perform a pressure hold test: pressurize the loop to 1.5x the operating pressure using a hand pump or the CDU's own pump with a bypass valve. Close all isolation valves and record the pressure every 15 minutes for 24 hours using a calibrated pressure transducer and data logger. The pressure must not drop more than 0.1 bar over the test period; any drop indicates a leak. If a leak is suspected, isolate sections by closing rack manifold valves and re-pressurizing each section individually to locate the source. After the test, bleed the loop again to remove any air that came out of solution during pressurization, and verify the coolant's pH (typically 7-8.5) and conductivity (under 10 µS/cm) with a handheld meter. Record these values as part of the baseline.
Flow Balancing and Temperature Setpoint Configuration
With the CDU running at its design flow rate (e.g., 100 L/min for a 300 kW unit), use a clamp-on ultrasonic flow meter at each rack's supply line to measure actual flow. Adjust the balancing valves on each rack's manifold until all racks receive within 10% of their target flow — for example, 10 L/min per rack for a 30-rack row. If one rack is low, open its valve slightly and close another's to redistribute without changing the total pump speed. Record the final flow readings and label each valve position with a durable tag. The target flow per rack is calculated from the total GPU power (kW) divided by the coolant's specific heat capacity and the desired temperature delta (typically 10°C).
Set the CDU's supply temperature to the OEM-recommended value (typically 22°C for H100/B200 direct-to-chip cooling) via the CDU's controller. The controller should modulate the facility water control valve (a 3-way or 2-way modulating valve) to maintain this setpoint within ±1°C. Verify the return temperature at the CDU's inlet; it should be 5-10°C higher under full load. If the delta is too low, the flow is too high (pump energy waste); if too high, the flow is too low (risk of hot spots). Adjust the pump speed or balancing valves to achieve a delta within the OEM spec. Enable the CDU's alarm thresholds: high supply temp at 30°C (or 5°C above setpoint), low flow at 80% of setpoint, and high pressure at 5 bar (or 1.2x operating pressure).
Redundancy Configuration and Failover Testing
For N+1 redundancy, install two CDUs in parallel, each sized for the full load, with motorized isolation valves (normally open for primary, normally closed for standby) on the primary-side supply and return lines of each CDU. Each CDU must be on a dedicated power feed from separate UPS sources to protect against electrical failure. The CDUs' controllers must be networked via Modbus or BACnet to a master controller that monitors both units and switches to the standby if the primary fails. Configure the standby CDU's pump to run at a low speed (10% of max) with its isolation valves slightly open to keep the coolant warm and prevent thermal shock on failover. Set the failover threshold: if the primary's supply temperature exceeds the setpoint by 3°C for more than 10 seconds, or if its flow drops below 50% for 5 seconds, the master controller opens the standby's isolation valves and ramps its pump to full speed within 30 seconds.
Test the failover by simulating a primary CDU failure: close its motorized isolation valves (or trip its breaker) and monitor the standby's ramp-up time and the temperature rise at the rack inlets. The temperature should not exceed the GPU's maximum inlet spec (typically 35°C) during the transition. Record the failover time and temperature delta; the standby should reach full flow within 30 seconds. Repeat the test with the standby as primary to verify symmetry. Document the failover logic and test results in the commissioning report. For 2N redundancy, each CDU serves half the load while the other half is redundant; test by shutting down one CDU and verifying the remaining unit can handle the full load without exceeding its flow or temperature limits.
Common Failure Modes and Field Diagnostics
The most common failure is a coolant leak at a quick-disconnect fitting on a GPU cold plate, often caused by an O-ring that was pinched during installation or a fitting that was not fully seated. Catch this by performing the 24-hour pressure test before powering GPUs, and by using a thermal camera during the first 24 hours of operation to spot cold spots from evaporative cooling at a leak. If a leak is suspected, isolate the rack and retorque all fittings; if the leak persists, replace the O-ring and inspect the mating surface for scratches. Another frequent issue is air entrapment in the loop, which causes flow fluctuations and temperature spikes. Diagnose it by listening for gurgling sounds at the rack manifolds and by observing erratic flow meter readings. Purge the air by running the pump at full speed and opening the highest vent valve for 30 seconds while watching the flow meter stabilize.
A third failure is pump cavitation from a clogged inlet filter or low coolant level. Check the CDU's pressure differential across the pump; if it drops below the OEM spec, inspect and clean the filter. Also verify the coolant level in the sight glass; if it's low, top off and check for leaks. Finally, a control valve failure on the secondary side can cause the supply temperature to drift. Monitor the valve's position feedback; if it is fully open and the supply temp is still rising, the facility water is too warm (above setpoint minus 5°C) or the heat exchanger is fouled. Clean the heat exchanger plates per the OEM procedure (typically a chemical flush with a descaling solution followed by a deionized water rinse) and verify the facility water temperature is within spec. If the valve fails to open, check the actuator linkage and control signal.
Baseline Documentation and Handover
After commissioning, create a baseline document that includes: the CDU's serial number, firmware version, and all setpoints (supply temperature, flow, pressure, alarm thresholds). Record the flow readings for each rack, the pressure drop across the CDU's heat exchanger (at design flow), and the coolant's pH and conductivity. Take a photo of the CDU's controller screen showing the live values. This baseline is used for future troubleshooting and to detect drift (e.g., increasing pressure drop indicates heat exchanger fouling). Label every valve, sensor, and connection with a durable tag that matches the as-built diagram, and store the diagram in the facility's asset management system.
Hand over the commissioning report to the facility's operations team, including the 24-hour pressure test log, failover test results, and a list of spare parts (e.g., O-rings, filters, gaskets, valve actuators) with part numbers. Train the team on how to reset alarms, change filters, and top off coolant — demonstrate the procedure on the live system. Provide a contact for Leviathan Systems for any follow-up support. The report should be stored in the facility's asset management system and updated after any maintenance event. Ensure the team knows how to access the CDU's web interface and interpret the trend logs.
Standards referenced: ASHRAE TC 9.9 (Liquid Cooling Guidelines for Data Centers) · ISO 4406 (Cleanliness Code for Hydraulic Fluids) · IEC 62443 (Industrial Communication Networks Security) · NFPA 75 (Standard for the Protection of Information Technology Equipment) · OEM-specific CDU installation manuals (follow the manufacturer’s required torque specs, fill rates, and alarm thresholds)
Frequently asked_
What is the correct procedure for filling a CDU loop to avoid air pockets?
Fill from the lowest point in the loop — typically the CDU's reservoir — using a fill cart with a 5-micron filter. Open all vent valves at the highest points in each rack's manifold. Fill slowly at under 10 L/min to prevent turbulent mixing that introduces micro-bubbles. After filling, run the pump at low speed (20-30% of max) for 30 minutes to circulate and degas, then increase to full speed for another 30 minutes while monitoring the reservoir level and topping off as needed. Repeat the venting process after the pressure test to release any air that came out of solution.
How do I verify that the CDU's flow rate is sufficient for the GPU load?
Use a clamp-on ultrasonic flow meter at each rack's supply line and compare the reading to the target flow per rack, calculated from the total GPU power (kW) divided by the coolant's specific heat capacity and the desired temperature delta (e.g., 10°C). The OEM typically specifies a minimum flow per GPU cold plate (e.g., 1-2 L/min for H100/B200-class GPUs). If the measured flow is below the target, adjust balancing valves or increase pump speed. Also check the CDU's return temperature delta; if it exceeds the OEM spec (usually >10°C), flow is insufficient and may cause hot spots.
What are the critical alarm thresholds I should set on the CDU controller?
Set a high supply temperature alarm at 30°C (or 5°C above the setpoint), a low flow alarm at 80% of the setpoint flow, a high pressure alarm at 5 bar (or 1.2x the operating pressure), and a low coolant level alarm at 20% of the reservoir capacity. Also enable leak detection alarms from the sensors under each fitting. Configure the controller to send alerts via SNMP or email to the facility's monitoring system, and to automatically shut down the CDU (or close a bypass valve) if the supply temperature exceeds 35°C to prevent GPU damage.
How do I test CDU failover in an N+1 configuration without disrupting GPU operation?
First, ensure the standby CDU is running at low speed (10% of max) with its motorized isolation valves open (or slightly open). Simulate a primary failure by closing the primary's isolation valves (or tripping its breaker) and monitor the standby's ramp-up time and the temperature at the rack inlets. The standby should reach full flow within 30 seconds, and the rack inlet temperature should not exceed the GPU's maximum spec (typically 35°C). Perform this test during a maintenance window when GPUs are idle or at low load. Repeat with the standby as primary to verify symmetry. Document the failover time and temperature delta.
What should I do if the CDU's supply temperature drifts above the setpoint?
First, check the facility water supply temperature; if it is above the CDU's setpoint minus 5°C, the heat exchanger cannot reject heat effectively. Verify the facility water flow rate and pressure. Next, inspect the CDU's control valve position; if it is fully open and the supply temp is still rising, the valve may be stuck or the heat exchanger may be fouled. Clean the heat exchanger plates per the OEM procedure (typically a chemical flush with a descaling agent followed by deionized water rinse). Also verify the coolant level and pump speed; low flow through the heat exchanger reduces heat transfer. If the issue persists, check the temperature sensor calibration with a calibrated RTD and confirm the controller setpoint is not inadvertently changed.