D6 Phase 3: Data Center Innovation
December 18, 2008 by Kevin Hazard, Web Hosting Evangelist in Data Centers, Evangelist's Corner, The Planet
You’ve probably been on pins and needles since you read our last blog post … anxiously anticipating the inside information of how we built a better-than-ideal-efficiency data center.
If you’re familiar with data center design, you’ve probably heard the terms “cold aisle” and “hot aisle.” A cold aisle is an aisle between racks of servers that sends cool air up through the floor (in the case of raised flooring data centers) and into the fronts of the servers. The air cools the server components and is exhausted through the back of the server as warm air, creating a hot aisle behind the server. Data centers are typically set up with racks of servers arranged front-to-front and back-to-back so that for every 2 rows of servers, only one cold aisle is needed. (If you’d like to see this setup in action, check out our popular Data Centric post.)
D6 Phase 3 uses those principles in a different way: hot and cold air are completely isolated.
Let’s take a look at how the phase is built to see what that looks like and why it is much more efficient.
In a raised-floor data center, your air conditioner blows air down under the floor into an air-tight compartment, and you insert vented floor tiles in the areas you want to cool. The large black unit in the image below is one of this phase’s air conditioners.
The metal posts you see on the ground are the braces used to create the grids upon which flooring tiles are installed.
The photo below is the bottom of a flooring tile. You’ll notice that there are no screws or bolts on the tile … it is simply laid on top of a grid of braces to make an air-tight seal. Each of the tiles is partially made out of concrete and is relatively heavy, so when a tile is installed, it’s not going anywhere unless you really want it to.
Below, you can see what the flooring grid looks like without floor tiles installed.
As we continue building the flooring, the data center appears to be taking shape … pretty standard process up to this point. But now, take a look at the air conditioning units below. Notice anything strange (aside from the fact that the covers aren’t installed)?
The return air plenum extends all the way to the ceiling of the room … now we’re getting somewhere.
I only retained a few things from my elementary school physics lessons, but one of them was that warm air rises and cold air sinks. This natural phenomenon is used in data center cooling: the air conditioners send the cold air down under the floor to cool the servers, when the servers send out warm air, the warm air rises to the top of the room, and the air conditioners pull the warm air from the top of the room to process and send back down as cold air. As Jeff mentioned in his interview, the higher the air conditioner pulls the air from, the warmer that air will be, and you don’t want to cool already cold air, so you should try and pull the hottest air in the data center.
Phase 3 goes a step further: it creates an airtight space above the ceiling tiles where all the warm air is exhausted and pulled in by the air conditioners.
So once we’ve got the floor tiles and the ceiling tiles installed, our fresh data center phase (without any server racks) looks like this:
This begs a big question: if the cold air is being sent through the floor to the servers and the warm air is being pulled from the ceiling, how do the servers pull the cold air and push the warm air without the heat being disseminated into the other areas of the data center?
Enter our new custom-made rack-mount cabinets.
The cabinet above is pulled out in one of our other data centers to demonstrate the design of the Phase 3 cabinets. The servers will pull cold air from the floor in front of the server but instead of exhausting the warm air out into an open aisle, the warm air will rise through a vent sealed to the ceiling (the sides, front and back of the cabinet are closed when the covers are installed).
So a completely installed server row looks like this:
One noticeable difference between this data center and the other data centers we’ve shown you in Houston and Dallas are the servers themselves: all rack-mount, no towers.
The rack-mount servers allow for a better power density throughout the data center, so we’ve got to make sure we can provide the power to all of our new servers in the event of a utility power outage.
We’ve got N+1 power redundancy, so for every phase, we have a dedicated backup generator, and for every data center we’ve got an extra backup generator in case any of the other generators fail. New phase = new generator. It’s pretty interesting to see the generator without its skin, right?
Now that power is accounted for, we can install our PDUs for each aisle and prepare to get servers up and running in the new phase.
A little networking and wiring, and D6 Phase 3 is ready for business! In the first picture below, you can see the orange tubes which are primary network drops into the phase … and yes, even those are sealed in the ceiling.
Head over to our The Planet’s Flickr page for a few more pictures of the new data center phase.
-Kevin
December 18th, 2008 at 2:01 pm
Have you taken any measurements to discover what the temperature differential is from the bottom of the rack when air is entering (presumably in the 60’s) and when it leaves the top of a fully populated rack? The only part I can’t get my head around is how the servers at rack units 41, 42 aren’t getting considerably hotter air as their input, as it’s already been through 40U’s worth of equipment?
December 18th, 2008 at 2:28 pm
Very good question, Doug …
The key is that all of the hot air is sealed away from the cold air wherever possible. I know that doesn’t sound like it answers your question, but I’ll explain a little more:
The servers are all pulling air from the front of the chassis (or the part exposed to the aisle), and the air conditioning from under the floor is being vented out in front of the servers, rather than into the sealed enclosure. You’re right that the air inside the top of the cabinet will be hotter than the air lower down in the cabinet, but the servers at the top are still pulling air from the aisle which is only marginally warmer than the air at the floor … The key is that nothing is exhausting heat into the aisle to make the air being pulled by the higher server significantly warmer than air being pulled by the lower server.
Because the servers are designed a one-way air flow, the heat build up behind the server doesn’t really affect the performance of the server itself … it’s all about the air that is being pulled in from the front, since that’s what is really cooling the system.
December 18th, 2008 at 2:35 pm
so what is custom about the CPI cabinets? Same as Intel uses……
December 18th, 2008 at 2:49 pm
Thanks! That part wasn’t 100% clear to me from the descriptions or the photos – I couldn’t visibly see how any cool airflow was getting from under the floor to the intakes on the front of the servers. The finished racks pictured didn’t appear to have any ventilation in the floor panels in front of them – so it was a bit of a mystery. Thanks for clearing it up!
December 18th, 2008 at 2:54 pm
I put my elementary graphics skills to the test to create an airflow diagram that helps show what’s going on: http://blog.theplanet.com/images/airflow.png.
December 18th, 2008 at 3:14 pm
Bob, That’s a good catch … the cabinets themselves are relatively standard, and we’re definitely not the first to use Hot Air Isolation techniques to improve data center efficiency … when it came to implementation, though, the sealed chimney at the top of the cabinet wasn’t exactly a “buy off the rack” option, given the size of the data center space (though we worked with our vendors to get them as standard as possible).
Doug, looking around a bit more, your confusion is absolutely justified and I wasn’t entirely correct: One of the benefits of using hot air isolation is that you don’t have to install ventilated tiles in front of every server rack … you just have to cool the room as a whole and the servers will pull in that cool air. The ventilated floor tiles are very effective in our other data centers because they focus the airflow in front of the server (basically pushing through the warmer air up to the front of the servers) … when there isn’t any warm air circulating in front of the servers, it isn’t as imperative to focus the ventilation on a per-rack basis.
December 18th, 2008 at 4:35 pm
Thanks for the interesting writeup. I have some questions, if you have a chance to address them:
–what happens if there’s a fire in one of the cabinets?
–any local fire code issues that needed variance?
–how did you decide how high to make the raised floor?
–the horizontal network runs are in trays in the cold air plenum? The orange innerduct in the photo for the vertical cable management runs to open trays?
Thanks.
December 18th, 2008 at 5:00 pm
Doug –
Another advantage of completely isolating the cold and hot air is that we can actually raise the set points on the CRAC units. You guessed that the air going into the servers was in the 60s, but in fact, it’s much warmer.
In a more typical datacenter, you have to make the AC units blast out really cold air, because it’s going to mix with hot air, and you want it to average out to something that a server is comfortable with. If there’s no hot air to mix with, you can set your AC unit to pump out air that’s maybe in the 70s. My guess is that a server wouldn’t even mind air in the 80s. I don’t know what the set points we’re using are, but the room is not meatlocker cold, because it just doesn’t need to be.
As far as vented tiles, they’re in there, but the vents didn’t photograph very well.
December 19th, 2008 at 7:07 am
Hi Kevin,
Excellent article! And a really nice datacenter, I’m jealous
I am wondering though why you guys put the Cisco switches like you did. Does this provide better cooling for them? I am currently having some cooling issues in a Cold Corridor solution where my switch is facing its front at the back (easier for cabling).
Thanks,
Peter
December 19th, 2008 at 3:41 pm
–What happens if there’s a fire in one of the cabinets?
The smoke would be detected by the detectors in the crac units … It is extremely rare to have a fir in a computer room, but if it happened, it would likely be contained in the cabinet as there is really not much combustible material in the room.
–Any local fire code issues that needed variance?
No variance was required, the dry pipe sprinkler system was designed to accommodate the exhaust ducts on the cabinets.
–How did you decide how high to make the raised floor?
The floor height is based largely around the power density of the room. Generally and up to a point, the higher the floor the better, regardless of power density.
December 22nd, 2008 at 3:33 am
I’m curious if you have any metrics as to how much you’re saving in cooling energy costs by isolating your exhaust? I’m also curious about your exhaust mechanism. You mention plenum, but it’s unclear from the pictures if there is actually plenum in the ceiling, or if you are just pressurizing the drop ceiling?
December 22nd, 2008 at 11:13 am
The hot air isolation design brings the phase’s coefficient of efficiency (the total power necessary to operate a data center divided by the power necessary to operate the servers alone) down to 1.5, so it’s better than the “ideal” rating of 1.6.
The plenum we’re talking about it is the pressurized return air channel on the top of each of the air conditioning units. Because the warm air naturally rises through the chimneys attached to the sealed cabinets, we don’t need an active exhaust system to push or pull the warm air out. The space above the ceiling is pressurized insofar as it is sealed to the pressurized return air plenums on each of the air conditioners, so the recycling of the warm air is uninterrupted after it rises from the cabinets.
January 1st, 2009 at 10:46 pm
Wouldn’t you gain efficiency by pulling air from the outside when it is colder than the hot air exhaust?
Have you thought of dumping heat into a lake, the ocean, or simply underground? Or maybe some industries would pay to get some stuff at least pre-heated. Here in Canada lots of people pay to heat their houses during winter
Right now it is -17C (near zero Fahrenheit) in Montreal, an hour north of the US border.
January 5th, 2009 at 12:50 pm
@Nicolas If you want to run a hose down to Dallas, I’ll see if I can hook it into an air vent for you to get you a little authentic Texas server heat piped into your house up there.
Creating a system that would accommodate a whole lot of extra complexity, but it would likely be a bit more cost-prohibitive. Also, since the airflow can essentially be standardized in the current setup (i.e. The air will always be XX degrees when it leaves the servers, it will always be XX degrees when it enters the A/C unit, it will always be XX degrees when it leaves the A/C unit and it will always be XX degrees when it enters the front of the servers) we can tweak the settings at each point to make everything work together as seamlessly as possible.
@Dr.Best Thanks! We’ve been around the industry for quite a while, and we’ve got a bit of experience under our belts, so it’s especially fun to share some of the new technologies we are working on.
January 5th, 2009 at 12:55 pm
@Peter That’s a great question, and it’s one that makes a lot of sense … I’m not sure what played into the arrangement of of the switch in the rack, but I’ll ask around and see if I can get any answers.
January 8th, 2009 at 6:00 am
@Kevin
Thanks for that. I am really curious about this and I’m sure I’m not alone in this.
January 23rd, 2009 at 10:26 am
Hi Kevin,
Thanks for sharing that, you must be proud of that design. I read where the chimney’s on the racks were custom. Are you saying that you picked a rack from someone and then had an HVAC company fabricate the chimney’s for them, and then same question for the tie-in to the return air ducting – was that all custom with the HVAC company?
Thanks,
Patrick
January 23rd, 2009 at 11:09 am
Hey Patrick,
The chimneys didn’t require us to fabricate a brand new rack-related product … they came from the same company that sources our cabinets for the phase. They are custom in the sense that they needed an airtight fit when they were installed. Similarly, the return air plenum at the top of the HVAC units had to be constructed and installed precisely to make the design work, but the design and installation of the plenums was independent of the chimneys at the top of the rack.
Thanks for the question! If you’ve got any more, keep ‘em coming!
-Kevin
February 8th, 2009 at 11:31 pm
How do you handle horizontal cable distribution between/among cabinets/racks, especially changes?
Most sites I’ve seen lately have a 3 tier system with fiber on top in ducts, 1 foot or so below is copper signaling, bottom tier is DC power distribution. In your model, these would all be in above the ceiling in the plenum, from what I can tell.
February 20th, 2009 at 12:03 pm
[...] a recent walkthrough of D6 Phase 3, I toted along a video camera to give you a first-hand look at what the data center looks like in [...]
May 28th, 2009 at 10:56 am
Hello There,
Nice works, but I want to ask (as an Electrical Engineer) what is the UPS system you installed and what is the power of it (kVA).
Regards,
September 23rd, 2009 at 5:02 pm
[...] our H2 data center, we only deployed racks of tower servers, and in our newest data center phase, D6 Phase 3, we only provision rack-mount servers. You might assume this shift to imply the complete dominance [...]