Zephyrnet Logo

The Weird World of Liquid Cooling for Datacenters

Date:

When it comes to high-performance desktop PCs, particularly in the world of gaming, water cooling is popular and effective. However, in the world of datacenters, servers rely on traditional air cooling more often than not, in combination with huge AC systems that keep server rooms at the appropriate temperature.

However, datacenters can use water cooling, too! It just doesn’t always look quite how you’d expect.

Staying Cool

Cooling is of crucial importance to datacenters. Letting hardware get too hot increases failure rates and can even impact service availability. It also uses a huge amount of energy, with cooling accounting for up to 40% of energy use in the average datacenter. This flows into running costs, as well, as energy doesn’t come cheap.

Thus, any efficiency gains in cooling a datacenter can have a multitude of benefits. Outside of just improving reliability and cutting down on emissions through lower energy use, there are benefits to density, too. The more effective cooling available, the more servers and processing power that can be stuffed in a given footprint without running into overheating issues.

Water and liquid cooling techniques can potentially offer a step change in performance relative to traditional air cooling. This is due to the fact that air doesn’t have a great heat capacity compared to water or other special liquid coolants. It’s much easier to transfer a great quantity of heat into a liquid. In some jurisdictions, there is even talk of using the waste heat from datacenters to provide district heating, which is much easier with a source of hot liquid carrying waste heat vs. hot air.

However, liquid cooling comes with drawbacks, too. Leaks can damage electronics if not properly managed, and such systems typically come with added complexity versus running simple fans and air conditioning systems. Naturally, that improved cooling performance comes at a trade-off, else it would be the norm already.

Various Approaches

imageDanish company Asetek has experimented with direct water cooling of server hardware, while also exploring using the waste heat for district heating purposes. Credit: Asetek press release
” data-medium-file=”https://hackaday.com/wp-content/uploads/2022/06/tilslut-1-asetek-datacenter-plus-andre-s-eriksen.jpg?w=400″ data-large-file=”https://hackaday.com/wp-content/uploads/2022/06/tilslut-1-asetek-datacenter-plus-andre-s-eriksen.jpg?w=500″ loading=”lazy” class=”size-medium wp-image-539616″ src=”https://hackaday.com/wp-content/uploads/2022/06/tilslut-1-asetek-datacenter-plus-andre-s-eriksen.jpg?w=400″ alt width=”400″ height=”266″ srcset=”https://hackaday.com/wp-content/uploads/2022/06/tilslut-1-asetek-datacenter-plus-andre-s-eriksen.jpg 500w, https://hackaday.com/wp-content/uploads/2022/06/tilslut-1-asetek-datacenter-plus-andre-s-eriksen.jpg?resize=250,167 250w, https://hackaday.com/wp-content/uploads/2022/06/tilslut-1-asetek-datacenter-plus-andre-s-eriksen.jpg?resize=400,266 400w” sizes=”(max-width: 400px) 100vw, 400px”>
Danish company Asetek has experimented with direct water cooling of server hardware, while also exploring using the waste heat for district heating purposes. Credit: Asetek press release

The most obvious water-cooling approach for a datacenter would be to swap out fan coolers in servers for water blocks, and link up racks to water cooling circuits. This is achievable, with some companies offering direct-to-chip cooling blocks that can then be hooked into a broader liquid cooling loop in a supporting server rack. It’s the same theory as water cooling a desktop PC, replacing fans and heatsinks with water blocks instead. This method of directly water-cooling servers has the benefit that it can extract a lot of heat, with some claims as high as 80 kW per rack.

However, this approach comes with several drawbacks. It requires opening up and modifying servers prior to installation in the rack. This is undesirable for many operators, and any mistakes during installation can introduce defects that are costly to rectify in both time and equipment. Service and maintenance is also complicated by the need to break water cooling connections when removing servers, too, though this is assuaged somewhat by special “dripless” quick-connect fittings.

A less invasive method involves the use of regular air-cooled servers that are placed in special water-cooled racks. This method removes any need to modify server hardware. Instead, air-to-water heat exchangers mounted at the back of the server rack pick up the heat from the hot server exhaust and dump it in into the liquid coolant. The exhaust air is thus chilled and returns to the room, while the coolant carries the waste heat away. Rooftop cooling towers, like the ones pictured at the top of this article, can then be used to extract the heat from the coolant before it’s returned. It’s not as effective as directly capturing the heat from an on-chip waterblock, but claims are that such systems can extract up to 45 kW of heat per rack.

In addition to using unmodified hardware, the system cuts down on the danger of leaks significantly. Any leaks that happen will be in the back of the server rack, rather than directly on the server’s circuit boards. Additionally, systems typically run at negative pressure so air is sucked in from any holes or damaged tubes, rather than liquid being allowed to leak out.

imageMicrosoft famously ran an underwater datacenter in a sealed tube back in 2018. The experiment had several benefits over traditional designs that run on land. Credit: Microsoft
” data-medium-file=”https://hackaday.com/wp-content/uploads/2022/06/natick800.webp?w=400″ data-large-file=”https://hackaday.com/wp-content/uploads/2022/06/natick800.webp?w=800″ loading=”lazy” class=”size-medium wp-image-539615″ src=”https://hackaday.com/wp-content/uploads/2022/06/natick800.webp?w=400″ alt width=”400″ height=”225″ srcset=”https://hackaday.com/wp-content/uploads/2022/06/natick800.webp 800w, https://hackaday.com/wp-content/uploads/2022/06/natick800.webp?resize=250,141 250w, https://hackaday.com/wp-content/uploads/2022/06/natick800.webp?resize=400,225 400w” sizes=”(max-width: 400px) 100vw, 400px”>
Microsoft famously ran an underwater datacenter in a sealed tube back in 2018. The experiment had several benefits over traditional land-based datacenters. Credit: Microsoft

More extreme methods, exist, too. Microsoft made waves by running a fully-submerged datacenter off the coast of Scotland back in 2018. With a cluster of conventional servers installed in a watertight tube, heat was rejected to the surrounding waters which kept temperatures very stable. The project ran for two years, and found that the sealed atmosphere and low temperatures were likely responsible for an eight-fold increase in reliability. Project Natick, as it was known, also promised other benefits, such as reduced land costs from locating the hardware offshore.

Microsoft isn’t resting on its laurels, though, and has investigated even wilder concepts of late. The company has developed a two-phase immersion cooling tank for datacenter use. In this design, conventional servers are submerged in a proprietary liquid developed by 3M, which boils at a low temperature of just 50 C (122 F). As the server hardware heats up, the liquid heats up.  It sucks up huge amounts of energy in what is called the latent heat of vaporization, required for the liquid to boil. The gaseous coolant then reaches the condenser on the tank lid, turning back to liquid and raining back down on the servers below.

imageMicrosoft has been experimenting with dumping servers in a non-conductive liquid which cools the immersed hardware via a phase-change to gas. Credit: Microsoft
” data-medium-file=”https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?w=400″ data-large-file=”https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?w=800″ loading=”lazy” class=”wp-image-539614 size-large” src=”https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?w=800″ alt width=”800″ height=”450″ srcset=”https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg 1920w, https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?resize=250,141 250w, https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?resize=400,225 400w, https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?resize=800,450 800w, https://hackaday.com/wp-content/uploads/2022/06/datacenter-liquid-cooling-server-blade-immersion_1920x1080.jpg?resize=1536,864 1536w” sizes=”(max-width: 800px) 100vw, 800px”>
Microsoft has been experimenting with dumping servers in a non-conductive liquid which cools the immersed hardware via a phase change to gas. Note the bubbling liquid warmed by the heat of the servers. Credit: Microsoft

The immersion method makes for excellent heat transfer between the server hardware and the coolant. As a bonus, it doesn’t just cool down a small section of the CPU via a heatsink. Instead, the entire server is free to dump heat into the liquid. The hope is that this would allow an increase in hardware density in datacenters, as well as an increase in performance, as the high cooling capacity of the immersion method allows for better heat removal in a much smaller space.

Of course, it’s a complex and high-end solution that will take some time before it’s ready for the mainstream. Datacenter operators simply aren’t used to dunking their hardware in liquid, nor used to running them in sealed containers to allow such a system to work. It’s likely that there would also be some maintenance headaches, where immersion tanks would have to be switched off prior to opening them for physical service of the hardware inside.

As humanity continues to crave more computing power, and we strive to cut energy use and emissions, expect further developments in this space. Sheer competition itself is a big driver, too. Any company that can cut running costs, land use, and find more performance will have an advantage over its rivals in the marketplace. Expect watercooling systems to become more mainstream over time, and some of the whackier ideas to find purchase if their major benefits are worth all the hassle. It’s an exciting time to work in datacenter engineering, that much is for sure.

spot_img

Latest Intelligence

spot_img