Immersion cooling is supposed to be the next big thing. It is being heralded by vendors as the next stage in the evolution of the data center, the only way to achieve ultra-high-density racks, and the best solution to bring high-performance computing (HPC) to a much wider range of users and applications.

But for many data centers, adding immersion cooling to existing facilities represents a major challenge – in some cases it may even be a fool’s errand. How will cabling fare underwater? What about the formation of scum on equipment and wiring? How on earth do you do regular maintenance checks in a liquid environment? Will cranes be needed to pull equipment out of the liquid for inspection and repair, and will there be room for them in the data center? Such considerations are causing some to hold back on liquid cooling adoption for now, particularly for immersion cooling.  

For example, Ohio Supercomputer Center (OSC), a statewide HPC resource for industry, government, non-profits, and schools in Ohio, was an early adopter of immersion cooling as far back as the late eighties. That experiment didn’t last long and now it only uses direct liquid cooling (DLC) on one of its supercomputer clusters. It is a low-pressure, rack-based passive cold plate loop used on Dell PowerEdge C6420 servers to liquid cool Intel processors in each of the 256 CPU nodes. The system is managed by a stand-alone, central pumping CHx650 coolant distribution unit.

“We haven’t begun investigating alternative cooling technologies beyond DLC,” said Doug Johnson, Associate Director of the OSC data center. “Immersion and phase-change cooling look interesting, but we’re waiting on wider adoption before we spend too much time considering the technologies.”

Let’s investigate the strengths of immersion cooling and the difficulties it poses for the data center.   

Immersion Cooling Benefits

Immersion cooling may well be the way of the future. The physics are very much in its favor as water is a far better conductor of heat than air. It is more than 20 times more efficient in transferring heat than air – and if you take a specific volume of water flow over a hot component (known as the volumetric flow rate), water has a heat carrying capacity nearly 3,500 times that of air.

Mark Gallina, System Thermal & Mechanical Architect at Intel, said liquid cooling more “efficiently distributes heat over more convection surface area (radiator) than pure conduction, allowing for reduced fan speeds (better acoustics) or higher total power.”

That said, it has many hurdles to overcome.

Lack of Standards

Immersion cooling lacks formally agreed standards on what kind of fluids should be used. This is necessary as different chemicals deployed by OEMs within the immersion fluids may have a negative impact over time on the IT equipment they are cooling.

“The fluids used in immersion cooling need to be inert, not degrade insulation, remove glues holding labels on equipment and cables, or react with inks on motherboards, chips, and other components,” said Mike Andrea, CEO of Oper8 Global and Data Center Institute (DCI) board member.

Fluid Leakage and Disposal

What about leakage? Yes, OEMs of immersion cooling solutions will go to great lengths to prevent it. But water will find a way, to paraphrase a famous line from the original Jurassic Park movie. Leaks could have devastating consequences to other data center equipment. Complete failures of immersion systems could flood the entire facility. This raises serious insurance questions that must be answered. It is quite likely that insurers will express some reluctance to insure data centers that wish to implement immersion systems. Equipment vendors, too, will be hesitant to honor warranties for systems damaged due to immersion fluid leakage. Finally, fluids will reach end of life and must be disposed of if deemed to be hazardous. That adds more cost.   

Maintenance Problems

Most current data centers are ill-equipped to manage servers in an aquatic setting. Will staff need special wet suits and will they need to be trained and certified in diving? Alternatively, are cranes required above immersion systems to haul the servers out? Such issues need addressed in a way that doesn’t add complexity and expense.

“Placing IT equipment into immersion cooling containers probably requires a crane system – either overhead or mobile floor-based systems that move between the immersion tubs,” said Andrea. “This means space and careful 3D planning of the data halls that don’t restrict movement of the cranes.”

Space Constraints

Speaking of space, immersion cooling places plenty of constraints on data center design. Cranes need several feet of room above racks. Many data centers simply don’t have the room. Ceiling integrity is another challenge. Some will not be able to bear the weight of a crane hauling immersion systems around. Even without cranes, immersion racks are typically wider and deeper. Room reconfiguration is probably going to be called for. Inevitably, that space constraints alone will outlaw immersion solutions in many facilities.

Looking Ahead

Despite these concerns, count on those on the leading edge of HPC and data center design to push the technology to the limit. It may take years until the wrinkles are removed, standardized solutions emerge, industry standards are issued, and prices come down enough to facilitate broader utilization. It remains to be seen whether liquid cooling can overcome these barriers and eventually become the de-facto standard for data center cooling. More likely, it will become a niche approach that helps to boost rack density in high-end applications.

“Immersion cooling has the potential to take data center rack densities to whole new dimensions,” said Greg Schulz, an analyst with Server and StorageIO Group.