This is part 2 of a three part series on ASHRAE’s recommended vs allowable data center environmental limits. To read part 1, click here, and to read part 3, click here.

What really is the difference between the ASHRAE TC9.9 recommended data center environmental limits and the allowable limits? Conventional wisdom might say something like the recommended temperature envelope for server inlet air temperature is 17-27˚C (64.4˚F – 80.6˚F) and the allowable range is something wider than that. The problem with conventional wisdom is that it accepts the confusion inherent with the misapplication of these defining terms. On close examination, I would assert that the recommended limits allow businesses to frivolously waste capital and operating budgets, while the allowable limits ought to be recommended for businesses to honor their fiduciary responsibilities to all their stakeholders and moral responsibilities to future generations.

At this point, most of our readers are well aware of the latest ASHRAE guidelines for server inlet temperatures and are probably at least somewhat familiar with the “X” Factor, so my review will be very brief. After causing some confusion in the industry with the original release of the allowable ranges for server inlet temperatures that could be permitted for some undefined short periods of time, ASHRAE clarified matters some seven years ago by developing the “X” factor, which we could all use to individually determine what was an acceptable short period of time for exposure to the wider range of temperatures allowed for the different classes of servers (A2, A3, A4). The server OEMs on TC9.9 would not share specific information about their product’s failure rates, but they did share information on variances of failure rates based on inlet temperatures. They eventually agreed on 68˚F as the baseline figure, and then agreed on what increases or decreases in that base failure rate could be expected based on server inlet temperatures being either higher or lower than 68˚F. For example, at 59˚F we should expect to see about 72% of the failures we might expect to see at 68˚F and at 77˚F we might expect to see 124% of the failures we might typically see at 68˚F. They published a table with low range, mid range and high range values for each 2.5˚C bin above and below the 68˚F baseline “X”. The original paper in which the “X” factor was launched included a case study for Chicago that showed that a data center without any air conditioning, relying on free air cooling modulated by the whims of mother nature would actually see 1% fewer server failures than a data center that maintained a constant 68˚F server inlet temperature all year long.

In today’s piece I have developed various case studies for different environmental limits, different types of cooling economization and for fourteen different data center geographic locations. Tagged on to the back end of this piece are more data tables than you probably care to pore over, but they are available for you to scroll back and forth to as your curiosity so inclines you.

The first four tables establish free cooling availability in each of the fourteen locations for servers operating within the recommended range, within the A2 class allowable range, within the A3 class allowable range and within the OpenEdge server envelope. One quick caveat on the OpenEdge server specification. It cites a maximum 45˚C inlet temperature but allows for short excursions up to 55˚C. There is no definition for “short” and this equipment was not part of the “X” factor development, so there is not yet much analysis I can include. Nevertheless, at either 45˚C (essentially class A4 allowable) or 55˚C, the possibilities are definitely intriguing. The assumptions for these available free cooling hours include mandatory excellent airflow management in which there is no more than a 2˚F increase from where the supply air enters the data center to where it is ingested by any server in the room. Therefore, I have used a 2˚F “approach” temperature for direct airside economization, a 15˚ cumulative approach temperature from ambient wet bulb to delivered supply dry bulb for plate and frame water-side economization, 5˚F approach temperature for air-to-air heat exchanger economizer (such as a heat wheel), and 7˚ approach temperature from ambient wet bulb to delivered dry bulb for indirect evaporative cooling systems. The 2˚ room variation plus approach temperatures have been backed out of every calculation in every table. The main take away from these first four tables is that, with just a couple exceptions, some form of free cooling is available just about everywhere for most of the year.

In Tables 5 and 6 I have exposed my methodology for determining the variance from the “X” factor for server failures in a data center with no air conditioning. One table for my non-Celsius readers and one for the rest of the community. The server inlet temperature bins in the left columns were all factored by the associated approach temperatures before pulling hours from my weather data base. You will note in Omaha we have fewer server failures projected for air side economization and indirect evaporative cooling than we would expect to see at a constant 68˚ inlet temperature and in Sydney we have somewhat elevated failure rates in all four scenarios.

In Table 7 I have accumulated the summary lines for all fourteen locations from similar spreadsheets to tables 5 and 6 – just the answers without all the detail, but the process was the same. We see a pretty wide range of results, ranging from server failure rates near 80% of the baseline (20% less than constant 68˚F inlet temperature) in London, Frankfurt and Amsterdam up to 120-130% in some scenarios for Phoenix and Hong Kong, with most variations being of the single digit variety above and below the “X” factor baseline.

However, if you recall the first four tables, there were many data points for locations and economizer styles wherein there was not access to free cooling 100% of the time. So for example, if we consider Class A2 servers with a maximum allowable 35˚C (95˚F) inlet air temperature, we will be out of spec and perhaps in warranty violation in many of these situations. Therefore, in Table 8 I have shown how much air conditioning we would need to add to a 1MW data center in order to not exceed those allowable limits. While the point of the “X” factor is to show us a path to a data center with no mechanical cooling, as a practical matter that ideal is not always possible. However, since we only need to bump our free cooling supply down from, for example, 98˚F to 93˚F (remember our 2˚ “room approach”?), we do not need a full mechanical plant to accomplish that. Whereas 1MW of IT load at around a 20˚F ΔT is going to require 284 tons of cooling under normal circumstances, we only need a portion of that to compensate for occasional high free cooling temperatures. I looked at the maximum temperature (minus all the associated approach temperatures) for each data center location and calculated the ratio of air conditioning that would be required to bump our inlet temperatures down into allowable A2 range. The results range from 70 tons for 4 hours in Chicago for free air cooling up to 185 tons for 5095 hours for air-to-air heat exchange in Phoenix. All the zeros (and there are plenty of them) refer to situations where no air conditioning is required. Just for comparison purposes, Table 9 shows air conditioning requirements to maintain the recommended temperature envelope.

Table 10 is at the heart of our economic argument. This is a comparison of Table 8 and Table 9 – I have captured the savings for both capital investment and operating expenses for operating at the Class A2 allowable limit versus operating within the recommended envelope. The percentages are reductions,, not just a ratio. Therefore, the 75% capacity and 99% operating hours savings for Chicago for free air cooling means that our chiller plant only needs to be sized 25% of what is required to meet the recommended envelope and we will only need to run it for less than 1% of the hours required (with free cooling) to meet the recommended limit. You will note some zeros for indirect evaporative cooling for Denver and London. There were no savings there merely because no air conditioning was required to meet the recommended limit with indirect evaporative cooling.

Finally, a small dose of reality needs to be interjected with the “X” Factor itself. How many of us have been running our data center with a constant server inlet temperature of 68˚F? I know there are still data centers out there with 68˚F set points resulting in 50˚F supply air and server inlet temperatures ranging from 65-85˚F, but most of us have left those days behind. With good airflow management, in fact, most of us have tended to inch our supply temperatures up and we have got relatively comfortable with inlet temperatures in the mid to upper 70’s. Therefore, basing a server failure forecast on a 68˚F baseline will not align well with our actual experience. While most healthy data center are probably operating at higher temperatures, just to be cautious, I have moved the baseline up to 72˚F in Table 11 and recalculated the anticipated failure rates for the four different types of free cooling in the fourteen locations on that recalibrated baseline. For the 1MW load of the running example, I have assumed 1400 servers and a baseline of 42 failures per year, about 3%. That is on the high side of data from recent studies conducted by “What Can We Learn from Four Years of Data Center Hardware Failures?”* and “Backblaze Hard Drive Stats for 2017,”** but my error on the side of caution is your ultimate pleasant surprise. The blue-shaded cells show data for scenarios where there is no air conditioning – 100% free cooling. Most of the data points show actual improvements in server reliability operating within the A2 server class allowable envelope as compared to maintaining a constant 72˚F server inlet temperature all year. As for those scenarios that show an increase in server failures, the reader is still faced with the business decision of the value of a small amount of IT hardware versus the capital expense of the cooling mechanical infrastructure as well as the cost of operating and maintaining it.

After having completed this study, I would say that I would have to recommend operating within the allowable envelope and I suppose grudgingly allow businesses with nothing better to do with their assets to go ahead and operate within the recommended range. There are obviously other ancillary issues involved with operating our data center at higher temperatures which I will touch on next time. For now, however, it seems to me pretty clear that economics and server reliability make a good case for recommending allowable temperature limits.

*Guosai Wang, Lifei Zhang, Wei Xu, Tsingua University and BaiDu (290,000 failure tickets analyzed)
**Andy Klein, www.blackbaze.com, Feb. 1, 2018 (91,000 drives analyzed),

Direct Air –Side Economization Availability

Table 1: Direct Air Side Economization Availability for Various Locations and Temperature Ranges

Indirect Air-Side Economization Availability

Table 2: Air-Side Economization Availability for Different Server Classes and Various Data Center Locations

Water-Side Economization Availability

Table 3: Water-Side Economization Availability for Different Server Classes and Various Data Center Locations

Indirect Evaporative Cooling Availability

Table 4: Indirect Evaporative Cooling Economization Availability for Different Server Classes and Various Data Center Locations 

Omaha “X” Factor Server Failure Forecast

Table 5: Example “X”” Factor Server Failure Forecast for a Data Center with no Air Conditioning in Omaha (˚F)

Sydney, Australia “X” Factor Server Failure Forecast

Table 6: Example “X”” Factor Server Failure Forecast for a Data Center with no Air Conditioning in Sydney, Australia (˚C)

“X” Factor Predicted Server Failure Variance for Data Center with no Air Conditioning Compared to Data Center with Constant 68˚F Server Inlet Temperature

Table 7: “X” Factor Predicted Server Failure Variance for Data Center with no Air Conditioning Compared to Data Center with Constant 68˚F Server Inlet Temperature 

Air Conditioning Required to Meet A2 Maximum Inlet Temperature for 1MW IT Load in Conjunction with Various Types of Economizers

Table 8: Air Conditioning Required to Meet A2 Maximum Inlet Temperature for 1MW IT Load in Conjunction with Various Types of Economizers

Air Conditioning Required to Meet Recommended Maximum Inlet Temperature for 1MW IT Load in Conjunction with Various Types of Economizers

Table 9: Air Conditioning Required to Meet Recommended Maximum Inlet Temperature for 1MW IT Load in Conjunction with Various Types of Economizers

Capacity and Run Time Savings for Air Conditioning in Data Center at A2 Allowable Inlet Temperature versus Recommended Temperature

Table 10: Capacity and Run Time Savings for Air Conditioning in Data Center at A2 Allowable Inlet Temperature versus Recommended Temperature

NET CHANGE IN SERVER FAILURES IN DATA CENTER OPERATING WITHIN A2 RANGE VERSUS MAINTAINING A CONSTANT 72˚F INLET TEMPERATURE

Table 11: Net Change in Server Failures in a Data Center operating within A2 Allowable Range versus Maintaining a Constant 72˚F Server Inlet Temperature

This is part 2 of a three part series on ASHRAE’s recommended vs allowable data center environmental limits. To read part 1, click here, and to read part 3, click here.