The Planet Blog

One of the great aspects of being in a new operations role is the ability to take a look at policies and procedures with a fresh eye. As the new senior director of Data Center Operations at The Planet, that’s been one of my first priorities.

I have never been a fan of policies written by senior executives and then simply rolled out with a directive to follow. For any policy to be effectively implemented, I contend that the team that must follow these policies should have a hand in helping to craft the policies or at least understand why the policies were created. More importantly, these processes come together through the team, rather than from a single individual’s effort. The best method I have seen for this comes from the Six Sigma Continuous Process Improvement (CPI) methodology.

In our data centers, we have created a Continuous Process Review (CPR) team made up of technicians, supervisors and managers across multiple cities. This team is solely responsible for the review and creation of all operations policies, processes and procedures across our six data centers.

The results from our team’s work together have been outstanding. In the last two months, we have produced new and/or updated policies for operational functions like ticket prioritization, technical escalations and management escalations. Since all levels of the data center operations organization are represented, the response to the new policies has been fantastic.

I owe many thanks to all the people who have participated on this new CPR team. None of this could have been achieved without their hard work and dedication.

The Planet’s goal is to provide the best customer experience in the hosting industry, and the data center operations team plays a huge role in that experience. Our ability to adapt to the evolving hosting landscape and respond directly to customer feedback is instrumental in improving the way our data centers run, so if you have any suggestions with regard to our data center policies and procedures, please let me know.

-Scott

Surprise! I’ve got some data center pictures for you. I’m sure this post caught you completely off-guard, so I’ll give you a moment to let your heart rate settle. If you want to call all of your friends to let them know that another incredible picture post is now available, I will wait for that as well.

…

Everyone ready for me to proceed? All heart rates back in the normal 60-100 beats per minute range? I can’t promise that your heart won’t race again as you get a partially serious tour of one of our Houston Data Centers, but for right now, we are safe. A couple months ago, Brooke posted a note about our Virtual Data Center Tour, and in the comments, a “real life” data center tour was requested, so here we are. I’ll give you a more focused picture tour of the data center, and if THAT isn’t enough, I’ll dust off the transporter and start taking reservations.

This row is one of our “hot rows.” I could have taken a picture of the fronts of all of the servers with all of the pretty lights, but this side of the servers is what you really want to see from your data center. Every cable is individually marked and designated for a specific server on that rack.

See … I told you. Because the labeling and organization of each of the cords is harder to capture up-close with the larger boxes, this picture was taken behind one of our rack-mounted server racks. Don’t worry … I didn’t unplug boxes 2 and 3 on this rack to show more of the cords, but I can’t promise that I didn’t unplug ANYTHING …

This picture was taken immediately after I finished taking some crazy pills, and when I look at it, I can’t help but think of Mitch Hedberg’s “Bigfoot” observation:

I think Bigfoot is blurry, that’s the problem. It’s not the photographer’s fault. Bigfoot is blurry. And that’s extra scary to me, because there’s a large, out-of-focus monster roaming the countryside. Run. He’s fuzzy. Get outta here.

In this case, it was the photographer’s fault.

It can be argued that this picture isn’t really showing a whole lot, but I find the detail to cleanliness and organization extremely impressive. I had to recruit Aaron to give me a run-down on what each of the wires carries, and here’s the run-down:

• Pink: Local Switch to Regional Switch
• Yellow: Regional Switch to Distribution Router
• Blue: Primary Network
• Brown: Isolated Network
• White: Remote Reboot Unit to Remote Reboot Controller
• Orange: Out-of-Band Network
• Black: Environmental Sensor
• Purple: Lunar Module Recharge Antiquantifier

I’ll let you guess which one of those I made up.

This cage houses our transport equipment. You thought I was joking about the transporter in the second paragraph? Pshaw… I don’t joke around with Star Trek references. If you look closely at the top left of the picture, you can see an orange plastic protective sleeve; that’s where the “interwebs” come from. More technically, that tube is the “pipe” or transit fiber that connects our data centers to the World Wide Web.

This picture features some of our edge routers (in blue) and core routers (in black, to the right of the edge routers).

When I walked into the data center, I felt like I was in a Sierra Mist commercial. If I had to guess the temperature outside in Houston, I would guess that it’s in the 120-130 degree vicinity (highly exaggerated), but as soon as you open the 3rd or 4th locked door to enter the data center, you get a wave of 68-70 degrees (not exaggerated).

This is the third phase of this data center’s space. When I came to Houston a few months ago, this phase was just getting raised floor installed, and I was surprised to see that the floor was “raised” about three feet from the ground. What goes under the floor in that three foot gap? Well, magic and rainbows, of course … along with power cables and the under-floor A/C vents.

This room has a caution sign on its door: “ACID.” As I was walking through the data center, snapping these pictures, one of my coworkers (who shall remain nameless), saw the sign and in a quizzical tone said, “A. C. I. D.? … I wonder what that stands for.”

While I was trying to come up with a smart-ass response to poke fun at that question, he (or she) sheepishly realized that “A. C. I. D.” might actually spell “acid.” The picture you see above is in the Uninterrupted Power Source room… which happens to be filled with batteries which happen to have battery acid which happens to lend itself to a warning sign. If the data center loses power for any reason, the batteries in this room immediately shoulder the energy load while the generators start-up.

This picture — and the next picture — were taken in the power transfer switch room. This area is the brain deciding where power is coming from (outside electricity, battery or generator) and where it is going (which phase of the data center).

These are the power transfer switches for each of the three phases of this data center. Needless to say, I didn’t touch anything in this room.

Generators. Four of them. Big ones. With such a large data center requiring so much power (and so much redundancy), our power backup couldn’t be a hamster in a wheel or a lawn mower engine. To get a perspective of how large each of these generators is, please see the picture below where I shamelessly cameo again.

Now, I’m anywhere between 6′3″ and 6′9″ depending on which convenience store I’m leaving (joke reference: Ron White), so these generators each have to be around 18 feet tall and 10 feet wide.

This picture features one of our data center NOCs (Network Operations Centers). These folks monitor the data center and are the “hands” that work with our server hardware.

As PBS would say, these tours were made possible by viewers like you. If you’ve got any more requests for pictures or brilliant insight from me, don’t be shy about sharing them.

- Kevin