Back and by popular demand, I’ve got a new set of pictures for you! About a week ago, I was in Dallas, where I did a little investigatory photo-journalism in our offices and in a couple of our data centers. I could add a disclaimer on the picture quality, but if you’ve endured the last few picture-laden posts, you’re probably desensitized to the blurred focus and poor photo composition.

If you were to drive South on I-35E through Dallas, you’ll see one of The Planet’s offices on your right-hand side just before you got to the home of the Dallas Mavericks and Dallas Stars.

Trendy, contemporary, stylish, post modern? Office buildings like this one make me want to pretend to be an architect (like George Costanza).

Did you think the stylishness was going to be left with just the exterior? You should be embarrassed about being so wrong. On the right side of the hall, you will see a large white paper with hand-written notes. This poster was created at a recent meeting where every manager committed to improving some aspect of our business, and it is displayed prominently to remind our team of the goals we set.

As you can see, we don’t relegate the fluorescent pink ethernet cables to the data centers, but the keen observer will note that the cables didn’t make it too far away from a data center. This conference room (called Asteroids) overlooks a small portion of our D2 data center.

The window in the upper-left corner of this image is the conference room from the previous picture. D2 is one of The Planet’s legacy data centers. As you can see, the central floorspace is occupied predominantly by cages and cabinets with no overhead cabling. How do the cables get run? I thought you’d never ask …

In data center tours of this facility, these plexiglass sections are a big attraction. The cable tray is run under the floor to get the cables to their respective destinations. I mentioned how raised our “raised flooring” is in the “Data Centric” post, but I snapped a perspective picture for you from D6:

Before you cry “Foul!” on this picture, I should explain that the Dr. Pepper bottle in this picture was not opened in the data center … it’s simply one of the most standard heights I had to give a good perspective of the floor’s height.

I am starting to believe that the bright colors used on the cabling in our data centers is meant specifically to keep me from taking a clear picture of a hot row.

I realized after the previous data center post that I didn’t include any pictures of the other side of a server row, so I made sure to include one in this post. Aren’t the blue lights soothing? If you walk down this aisle, you feel like you need to be listening to Blue Train (which happens to be the album playing through my headphones right now).

Here is another beautiful picture of the side of the server that really matters. This Clovertown rack again features the organization I explained previously: every cable is individually numbered and assigned to a specific box to make server identification as quick and easy as possible.

Cut so short? No!!! Well, the mark of a great storyteller is the ability to leave an audience wanting more, so while I have a few more pictures of the backbones of the data centers (power, generators, network, GNOC, etc.), I’ll save those for a rainy day.

- Kevin

Surprise! I’ve got some data center pictures for you. I’m sure this post caught you completely off-guard, so I’ll give you a moment to let your heart rate settle. If you want to call all of your friends to let them know that another incredible picture post is now available, I will wait for that as well.

…

Everyone ready for me to proceed? All heart rates back in the normal 60-100 beats per minute range? I can’t promise that your heart won’t race again as you get a partially serious tour of one of our Houston Data Centers, but for right now, we are safe. A couple months ago, Brooke posted a note about our Virtual Data Center Tour, and in the comments, a “real life” data center tour was requested, so here we are. I’ll give you a more focused picture tour of the data center, and if THAT isn’t enough, I’ll dust off the transporter and start taking reservations.

This row is one of our “hot rows.” I could have taken a picture of the fronts of all of the servers with all of the pretty lights, but this side of the servers is what you really want to see from your data center. Every cable is individually marked and designated for a specific server on that rack.

See … I told you. Because the labeling and organization of each of the cords is harder to capture up-close with the larger boxes, this picture was taken behind one of our rack-mounted server racks. Don’t worry … I didn’t unplug boxes 2 and 3 on this rack to show more of the cords, but I can’t promise that I didn’t unplug ANYTHING …

This picture was taken immediately after I finished taking some crazy pills, and when I look at it, I can’t help but think of Mitch Hedberg’s “Bigfoot” observation:

I think Bigfoot is blurry, that’s the problem. It’s not the photographer’s fault. Bigfoot is blurry. And that’s extra scary to me, because there’s a large, out-of-focus monster roaming the countryside. Run. He’s fuzzy. Get outta here.

In this case, it was the photographer’s fault.

It can be argued that this picture isn’t really showing a whole lot, but I find the detail to cleanliness and organization extremely impressive. I had to recruit Aaron to give me a run-down on what each of the wires carries, and here’s the run-down:

• Pink: Local Switch to Regional Switch
• Yellow: Regional Switch to Distribution Router
• Blue: Primary Network
• Brown: Isolated Network
• White: Remote Reboot Unit to Remote Reboot Controller
• Orange: Out-of-Band Network
• Black: Environmental Sensor
• Purple: Lunar Module Recharge Antiquantifier

I’ll let you guess which one of those I made up.

This cage houses our transport equipment. You thought I was joking about the transporter in the second paragraph? Pshaw… I don’t joke around with Star Trek references. If you look closely at the top left of the picture, you can see an orange plastic protective sleeve; that’s where the “interwebs” come from. More technically, that tube is the “pipe” or transit fiber that connects our data centers to the World Wide Web.

This picture features some of our edge routers (in blue) and core routers (in black, to the right of the edge routers).

When I walked into the data center, I felt like I was in a Sierra Mist commercial. If I had to guess the temperature outside in Houston, I would guess that it’s in the 120-130 degree vicinity (highly exaggerated), but as soon as you open the 3rd or 4th locked door to enter the data center, you get a wave of 68-70 degrees (not exaggerated).

This is the third phase of this data center’s space. When I came to Houston a few months ago, this phase was just getting raised floor installed, and I was surprised to see that the floor was “raised” about three feet from the ground. What goes under the floor in that three foot gap? Well, magic and rainbows, of course … along with power cables and the under-floor A/C vents.

This room has a caution sign on its door: “ACID.” As I was walking through the data center, snapping these pictures, one of my coworkers (who shall remain nameless), saw the sign and in a quizzical tone said, “A. C. I. D.? … I wonder what that stands for.”

While I was trying to come up with a smart-ass response to poke fun at that question, he (or she) sheepishly realized that “A. C. I. D.” might actually spell “acid.” The picture you see above is in the Uninterrupted Power Source room… which happens to be filled with batteries which happen to have battery acid which happens to lend itself to a warning sign. If the data center loses power for any reason, the batteries in this room immediately shoulder the energy load while the generators start-up.

This picture — and the next picture — were taken in the power transfer switch room. This area is the brain deciding where power is coming from (outside electricity, battery or generator) and where it is going (which phase of the data center).

These are the power transfer switches for each of the three phases of this data center. Needless to say, I didn’t touch anything in this room.

Generators. Four of them. Big ones. With such a large data center requiring so much power (and so much redundancy), our power backup couldn’t be a hamster in a wheel or a lawn mower engine. To get a perspective of how large each of these generators is, please see the picture below where I shamelessly cameo again.

Now, I’m anywhere between 6′3″ and 6′9″ depending on which convenience store I’m leaving (joke reference: Ron White), so these generators each have to be around 18 feet tall and 10 feet wide.

This picture features one of our data center NOCs (Network Operations Centers). These folks monitor the data center and are the “hands” that work with our server hardware.

As PBS would say, these tours were made possible by viewers like you. If you’ve got any more requests for pictures or brilliant insight from me, don’t be shy about sharing them.

- Kevin

Before I get into the meat of this entry, I feel it is necessary to address the comment made by my esteemed friend and colleague Thanh Tran, general manager of our Houston operations. In his blog entry earlier this month, Thanh mentioned “I work with a bunch of geeks. But that’s ok, because I’m a geek myself.”

I feel that this might have been a light-hearted jab at my previous entry, which I will readily admit was steeped in a healthy dose of science geekery.

Now, Thanh says that he too is a “geek.” For the record, I would like to point out that keeping a MS Paint-drawn Buzz Lightyear image file archived on a hard drive somewhere for nearly ten years pretty much seals the deal on one’s level of geekishness.

Thanh, I don’t think you had to tell us you are a geek; consider it self-evidenced. Just kidding around, of course …

All kidding aside, I’ve been looking forward to this blog entry, because I wanted to take a moment to acquaint everyone with just a few of the many outstanding data center technicians that our customers have probably worked with in the past. The next time you are reading a support ticket, check the usernames of the employees working the issue with you, and see if you recognize any of these excellent technicians.

For example, Jac has been with The Planet for more than two years, and is currently a Level 2 data center technician. A few days ago I was reading a recent ticket that he was working on for a customer who was suffering from a bit of “new server syndrome.” That’s the critical window of time after someone orders a new server that we have all experienced at one time or another. The customer was hard at work attempting to get his configurations in place so the server could be usable for his business, but he was running into some pretty wicked permission errors that he simply could not deduce.

Jac responded to the customer’s questions quickly and effectively, adjusting the settings to address the issues. He then went above and beyond expectations, providing the customer with some very detailed documentation related to these permission configuration issues. The problem was wholly solved, and the customer was quite happy. But more importantly, the customer was enabled to address his server needs more quickly in the future as well. It’s all about responding to business needs precisely and rapidly, and Jac Lindley does an excellent job of helping our customers achieve just that.

Another Level 2 data center technician I wanted to “introduce” is Johnny L., who has been with The Planet for almost three years, and in that time he has proven to be a strong anchor of knowledge in our data centers. Johnny is a whiz when it comes to server hardware; I’ve seen him swap out bad drives and start degraded array rebuilds so quickly it would make your head spin.

The pace of work in our data centers can be dizzying at times, and seeing guys like Johnny work can be awe-inspiring. But in the midst of all this activity, the thing that makes a guy like Johnny stand out is his attention to detail. Anyone can whip through a dozen unique hardware upgrades in record time, but not many can do it with the level of accuracy and care that Johnny displays.

I’m proud of all the technicians we have working in the data centers, but I thought to would be great to introduce a couple of our techs to everyone. Keep an eye out for them in a support ticket coming to an Orbit portal near you!

- Britt

Here at The Planet, I think a lot about our data centers since it’s my job to ensure that our servers are up and running 24×7x365. What keeps me up at night is the continuous power and cooling required to keep your servers and systems up and the preventive maintenance that’s required to run six world-class data centers. You could say I live and breathe data center maintenance. And with more than 44,000 dedicated servers depending on our infrastructure systems, it keeps me and my team hopping.

One of things we focus on is N+1 redundancy. It’s really a math equation that ultimately boils down to having backup systems in place so that we’re able to deliver a seamless experience for customers. We’re always working to improve the processes and equipment we have in place to maintain continuous uptime which is why we have redundant UPS (uninterrupted power supply), generator and HVAC systems in place.

Customer experience is important to all of us. That’s why we take N+1 redundancy so seriously. Preventive maintenance helps us ensure uptime. Maintenance and redundancy really go hand-in-hand. At The Planet, we offer N +1 redundancy of critical infrastructure components. This ensures that in case of a component failure, another system is in place and will automatically take over for the failed component. This provides a seamless transition from one component to another, and it’s completely invisible to customers and their servers.

Redundancy is also necessary to properly test and maintain these critical systems. There’s no way to fully and properly test and maintain them without redundancies, since we would run the risk of leaving customers exposed. During maintenance and testing we have to take these critical systems/components completely offline. It doesn’t matter whether it’s an HVAC unit or a UPS module or a generator. When a system is down without a redundant system in place, customers are exposed to the possibility of the in-use system failing with no back-up available. Now that’s a scary thought.

If you haven’t seen the data center tour on our Web site, I invite you to see what my team does. Click on the “Take the Tour” button at: http://www.theplanet.com/. I think you’ll be pleasantly surprised.

- Jeff

Managing operations in a data center can be pretty stressful, so one of my favorite ways to relax is reading scientific literature. I know it doesn’t sound like a very riveting hobby, but I’m rarely happier than when I have a copy of New Scientist or Scientific American in my hands. I consume the material with a voraciousness that might cause some to question my sanity. Mental health aside, I was reading an interesting article in Discover this month that provided one of those really great “ah ha” moments that related to The Planet’s data centers. Or more intimately, the place where your server lives.

The article covered some math that had been performed to determine the true, actual weight of the data that makes up the Internet. Starting with the weight of a single electron (2 x 10^-30 pound), the author broke down the number of electrons required to charge a single capacitor (the charge equaling a “1” in binary) in a computer’s memory (40,000), assuming a roughly 50 – 50 split on 1’s and 0’s in a typical 50 kilobyte e-mail. The resulting sum can then be used to determine an electron count per message (8 billion), landing us at a weight for a single e-mail of two ten-thousandths of a quadrillionth of an ounce. Now extrapolate that math across the whole of all Internet traffic; all the e-mail, Web pages, music, videos, instant messages and everything else we all contribute to the Internet. Data-wise you arrive at a mind-blowing 40 Petabyte number. However, that 40 Petabytes only equates to a weight of 1.3 x 10^-8 pound. That’s right … in real-world terms, all that data equals the weight of the smallest possible grain of sand, one measuring only two-thousandths of an inch across.

It’s pretty bewildering to think that all the effort and energy we contribute to this industry equates to such an infinitesimal true weight. Pondering that breakdown about the Internet, my thoughts drifted to another possible dissection: the data center. However, I wasn’t thinking in terms of the sub-atomic, but rather of the cellular. Let’s move from physics to biology.

Data centers are huge installations with myriad of moving parts that work to keep your server online and operational. So much so that you could liken the whole amalgamation to a living, breathing entity. Breaking down those parts to a cellular level proves to be an interesting exercise.

All organisms need sustenance, just as the data center needs AC power to survive. Think of the servers as cells in a body, varied infinitely, and specialized in both form and function. Striated muscle or peptidergic neural cells … MySQL database back-end or Battlefield 1942 server, the variations are vast. Communication is achieved via neurons arranged in a systemic network, just as the data center’s network interconnects dedicated servers to each other and to the outside world. Organelles that give metabolic function to a cell are comparable to RAM, hard drives and RAID cards, all of which are vital working parts of your server. You can even think of The Planet’s network security technicians as white blood cells fighting infection, and our DC technicians like powerful enzymes performing repairs on the cells. I guess you could refer to a RAM change out as an “organelle swap.”

But as anyone knows, the most important piece of any entity is its DNA: the core code that provides instruction, design and ultimately life to the organism. Paralleling that vital piece of the organism, a data center has DNA as well, which comes in the form of our customers. Without customers providing pattern and direction, the data center remains a nebulous blob of power, wiring and equipment without purpose or consciousness. But infused with our customers’ DNA, the data center springs to life, demonstrating utility and structure, ambition and organization.

So the next time that you are sweating a data migration or pushing code changes, stop to think about the part you play in this grand scheme and the amazing interrelation you have with this “data center organism.” But if that proves to be a bit too mind-expanding, then you can revert that tiny grain of sand that makes up all the information on the Internet. Large or small, we all have a part to play.