One of the great aspects of being in a new operations role is the ability to take a look at policies and procedures with a fresh eye. As the new senior director of Data Center Operations at The Planet, that’s been one of my first priorities.

I have never been a fan of policies written by senior executives and then simply rolled out with a directive to follow. For any policy to be effectively implemented, I contend that the team that must follow these policies should have a hand in helping to craft the policies or at least understand why the policies were created. More importantly, these processes come together through the team, rather than from a single individual’s effort. The best method I have seen for this comes from the Six Sigma Continuous Process Improvement (CPI) methodology.

In our data centers, we have created a Continuous Process Review (CPR) team made up of technicians, supervisors and managers across multiple cities. This team is solely responsible for the review and creation of all operations policies, processes and procedures across our six data centers.

The results from our team’s work together have been outstanding. In the last two months, we have produced new and/or updated policies for operational functions like ticket prioritization, technical escalations and management escalations. Since all levels of the data center operations organization are represented, the response to the new policies has been fantastic.

I owe many thanks to all the people who have participated on this new CPR team. None of this could have been achieved without their hard work and dedication.

The Planet’s goal is to provide the best customer experience in the hosting industry, and the data center operations team plays a huge role in that experience. Our ability to adapt to the evolving hosting landscape and respond directly to customer feedback is instrumental in improving the way our data centers run, so if you have any suggestions with regard to our data center policies and procedures, please let me know.

-Scott

Katie brought up some great points about an IT infrastructure provider checklist in her recent “World News from The Planet” video from ad:tech. Regardless of where you’re planning to host, you should always ask probing questions to determine just what, exactly, your provider brings to the table. Here are some points to consider:

Network

How does the provider’s network compare to the competition? Can it survive multiple bandwidth partner failures? Can it neutralize a multiple-Gbps DDOS attack?

Support

Is support staff available whenever I need it? Can I talk to a human immediately if I have an emergency? Does the provider employ certified staff that can help me with advanced issues?

Facilities

Can the provider host my IT infrastructure in multiple data centers and in more than one city? Does the company own its own facilities (or is it renting)? Is the data center entirely within the hosting provider’s control?

Power

Can the company survive a catastrophic power event such as being cut off from utility power for more than a week? Is the provider on a dedicated power grid within its facility or does it share a power infrastructure with other companies? Is the company’s power generation and distribution infrastructure regional (i.e., one unit per section of a data center) or is it integrated, providing true N+1 or better redundancy?

Vendors

Does the company use high-quality, name-brand vendors with equipment under warranty or does it offer generic, off-brand solutions? Do its vendors consider the company one of their top customers?

Product Portfolio

Does the provider offer a wide range of upgrades and add-ons or am I limited to its predefined packages? If the provider doesn’t have exactly what I need, can I special-order it?

Without definitive answers to questions like these, a price quote alone is pretty meaningless. After all, it’s the total value of the service — and not just the monthly fee — that is the true measure of any service offering.

Earth Day typically inspires widespread environmental introspection. How can we cut down on waste? Can we be more efficient? Are we actively pursuing “greener” operations? And how can we reduce our costs and be fiscally responsible?

Houston is recognized as the energy capital of the world, so it may be a surprise to learn that amidst that distinction The Planet does its part to reduce energy costs. In fact, we have been featured in several “green technology” articles over the past few months and acknowledged for our common sense approach. Ultimately, we look to save money, reduce consumption and improve data center efficiency. And in the coming weeks, we’ll announce an expanded program that takes us to the next level in increasing those efficiencies.

Tier1 is a leading research firm, and Martin Levy is the firm’s “green” analyst. In his report on The Planet, his headline was simple: “Down-to-earth solutions help improve efficiency at The Planet.”

Martin goes on to say the following:

Not a word about carbon offsets. Nobody planting trees. Nothing about Renewable Energy Credits (RECs). No recycling bins at the entrance to the datacenters. Instead, today’s announcement from The Planet was all about core datacenter efficiency. The company runs six datacenters and because of a focus on efficiency, it expects to save over one million dollars during 2008 … T1R is impressed. The Planet has shown that going green can be done the old-fashioned way. Make the technology work better and the company sees a positive ROI. That’s still good for the environment and even better for the bottom line!

Our facilities team is always on the lookout for new ways to reduce energy costs, since it’s one of our biggest expenses. Our vice president of facilities, Jeff Lowenberg, took an interesting challenge at the end of last year: Cut power costs by $1 million dollars in 2008, while we continue to grow and provision new servers in our six world-class data centers.

In his Sustainable IT blog, Ted Samson reported on a few of the initiatives aimed at improving our efficiency:

• Rearranging floor tiles to better manage cold airflow
• Installing seals and grommets in the ceilings, walls, and floors to reduce bypass airflow
• Installing blanking plates in server cabinets to direct airflow more efficiently
• Sealing power distribution units to reduce bypass airflow

Ted also explained the significance of those “minor” improvements:

Cool air was going to only where it was needed: the server intakes … Six months later, the company finds that its efforts have paid off substantially. Even though critical server loads increased by 5 percent, the facility’s overall cooling power needs dropped by 31 percent … The Planet also improved its “coefficient of efficiency,” an EPA- and Uptime Institute-recognized measurement of the total power necessary to operate a data center, divided by critical power, which represents the energy required to operate its computers. The company increased its rating to 1.7 – a near-ideal number – from its previous “good” ranking of 2.0.

Matt Stansberry at Search Data Center also spoke with Jeff about our progress and shared a few additional details in the quest to improve data center cooling:

Data center cooling is where most of infrastructure energy efficiency is lost. The fundamental rule in energy efficient cooling is to keep hot air and cold air separate … The Planet uses a method of extending the height of its computer room air conditioning (CRAC) units’ return-air plenums to optimize air cooling … By extending the plenums higher, it ensures that the CRAC units are not sucking in any cold air from the cold aisles, as it allows for the hottest air to be sucked into the units. In this scenario, the top of the plenums must be at least 2 feet from the ceiling.

To get an idea of what “plenums” are, you can visit Matt’s post or Heather Clancy’s recent article about The Planet at ZDNet’s GreenTech Pastures … and while you’re there, be sure to check out the post’s opening line.

To stay in the loop about what is being done in the “green tech” sphere, keep an eye on Ted Samson’s Sustainable IT blog, GreenerComputing, The Daily T1R from Tier1 Research, ZDNet’s GreenTech Pastures and Search Data Center.

And watch for more news from us.

-Yvonne

Back and by popular demand, I’ve got a new set of pictures for you! About a week ago, I was in Dallas, where I did a little investigatory photo-journalism in our offices and in a couple of our data centers. I could add a disclaimer on the picture quality, but if you’ve endured the last few picture-laden posts, you’re probably desensitized to the blurred focus and poor photo composition.

If you were to drive South on I-35E through Dallas, you’ll see one of The Planet’s offices on your right-hand side just before you got to the home of the Dallas Mavericks and Dallas Stars.

Trendy, contemporary, stylish, post modern? Office buildings like this one make me want to pretend to be an architect (like George Costanza).

Did you think the stylishness was going to be left with just the exterior? You should be embarrassed about being so wrong. On the right side of the hall, you will see a large white paper with hand-written notes. This poster was created at a recent meeting where every manager committed to improving some aspect of our business, and it is displayed prominently to remind our team of the goals we set.

As you can see, we don’t relegate the fluorescent pink ethernet cables to the data centers, but the keen observer will note that the cables didn’t make it too far away from a data center. This conference room (called Asteroids) overlooks a small portion of our D2 data center.

The window in the upper-left corner of this image is the conference room from the previous picture. D2 is one of The Planet’s legacy data centers. As you can see, the central floorspace is occupied predominantly by cages and cabinets with no overhead cabling. How do the cables get run? I thought you’d never ask …

In data center tours of this facility, these plexiglass sections are a big attraction. The cable tray is run under the floor to get the cables to their respective destinations. I mentioned how raised our “raised flooring” is in the “Data Centric” post, but I snapped a perspective picture for you from D6:

Before you cry “Foul!” on this picture, I should explain that the Dr. Pepper bottle in this picture was not opened in the data center … it’s simply one of the most standard heights I had to give a good perspective of the floor’s height.

I am starting to believe that the bright colors used on the cabling in our data centers is meant specifically to keep me from taking a clear picture of a hot row.

I realized after the previous data center post that I didn’t include any pictures of the other side of a server row, so I made sure to include one in this post. Aren’t the blue lights soothing? If you walk down this aisle, you feel like you need to be listening to Blue Train (which happens to be the album playing through my headphones right now).

Here is another beautiful picture of the side of the server that really matters. This Clovertown rack again features the organization I explained previously: every cable is individually numbered and assigned to a specific box to make server identification as quick and easy as possible.

Cut so short? No!!! Well, the mark of a great storyteller is the ability to leave an audience wanting more, so while I have a few more pictures of the backbones of the data centers (power, generators, network, GNOC, etc.), I’ll save those for a rainy day.

- Kevin

Surprise! I’ve got some data center pictures for you. I’m sure this post caught you completely off-guard, so I’ll give you a moment to let your heart rate settle. If you want to call all of your friends to let them know that another incredible picture post is now available, I will wait for that as well.

…

Everyone ready for me to proceed? All heart rates back in the normal 60-100 beats per minute range? I can’t promise that your heart won’t race again as you get a partially serious tour of one of our Houston Data Centers, but for right now, we are safe. A couple months ago, Brooke posted a note about our Virtual Data Center Tour, and in the comments, a “real life” data center tour was requested, so here we are. I’ll give you a more focused picture tour of the data center, and if THAT isn’t enough, I’ll dust off the transporter and start taking reservations.

This row is one of our “hot rows.” I could have taken a picture of the fronts of all of the servers with all of the pretty lights, but this side of the servers is what you really want to see from your data center. Every cable is individually marked and designated for a specific server on that rack.

See … I told you. Because the labeling and organization of each of the cords is harder to capture up-close with the larger boxes, this picture was taken behind one of our rack-mounted server racks. Don’t worry … I didn’t unplug boxes 2 and 3 on this rack to show more of the cords, but I can’t promise that I didn’t unplug ANYTHING …

This picture was taken immediately after I finished taking some crazy pills, and when I look at it, I can’t help but think of Mitch Hedberg’s “Bigfoot” observation:

I think Bigfoot is blurry, that’s the problem. It’s not the photographer’s fault. Bigfoot is blurry. And that’s extra scary to me, because there’s a large, out-of-focus monster roaming the countryside. Run. He’s fuzzy. Get outta here.

In this case, it was the photographer’s fault.

It can be argued that this picture isn’t really showing a whole lot, but I find the detail to cleanliness and organization extremely impressive. I had to recruit Aaron to give me a run-down on what each of the wires carries, and here’s the run-down:

• Pink: Local Switch to Regional Switch
• Yellow: Regional Switch to Distribution Router
• Blue: Primary Network
• Brown: Isolated Network
• White: Remote Reboot Unit to Remote Reboot Controller
• Orange: Out-of-Band Network
• Black: Environmental Sensor
• Purple: Lunar Module Recharge Antiquantifier

I’ll let you guess which one of those I made up.

This cage houses our transport equipment. You thought I was joking about the transporter in the second paragraph? Pshaw… I don’t joke around with Star Trek references. If you look closely at the top left of the picture, you can see an orange plastic protective sleeve; that’s where the “interwebs” come from. More technically, that tube is the “pipe” or transit fiber that connects our data centers to the World Wide Web.

This picture features some of our edge routers (in blue) and core routers (in black, to the right of the edge routers).

When I walked into the data center, I felt like I was in a Sierra Mist commercial. If I had to guess the temperature outside in Houston, I would guess that it’s in the 120-130 degree vicinity (highly exaggerated), but as soon as you open the 3rd or 4th locked door to enter the data center, you get a wave of 68-70 degrees (not exaggerated).

This is the third phase of this data center’s space. When I came to Houston a few months ago, this phase was just getting raised floor installed, and I was surprised to see that the floor was “raised” about three feet from the ground. What goes under the floor in that three foot gap? Well, magic and rainbows, of course … along with power cables and the under-floor A/C vents.

This room has a caution sign on its door: “ACID.” As I was walking through the data center, snapping these pictures, one of my coworkers (who shall remain nameless), saw the sign and in a quizzical tone said, “A. C. I. D.? … I wonder what that stands for.”

While I was trying to come up with a smart-ass response to poke fun at that question, he (or she) sheepishly realized that “A. C. I. D.” might actually spell “acid.” The picture you see above is in the Uninterrupted Power Source room… which happens to be filled with batteries which happen to have battery acid which happens to lend itself to a warning sign. If the data center loses power for any reason, the batteries in this room immediately shoulder the energy load while the generators start-up.

This picture — and the next picture — were taken in the power transfer switch room. This area is the brain deciding where power is coming from (outside electricity, battery or generator) and where it is going (which phase of the data center).

These are the power transfer switches for each of the three phases of this data center. Needless to say, I didn’t touch anything in this room.

Generators. Four of them. Big ones. With such a large data center requiring so much power (and so much redundancy), our power backup couldn’t be a hamster in a wheel or a lawn mower engine. To get a perspective of how large each of these generators is, please see the picture below where I shamelessly cameo again.

Now, I’m anywhere between 6′3″ and 6′9″ depending on which convenience store I’m leaving (joke reference: Ron White), so these generators each have to be around 18 feet tall and 10 feet wide.

This picture features one of our data center NOCs (Network Operations Centers). These folks monitor the data center and are the “hands” that work with our server hardware.

As PBS would say, these tours were made possible by viewers like you. If you’ve got any more requests for pictures or brilliant insight from me, don’t be shy about sharing them.

- Kevin

Before I get into the meat of this entry, I feel it is necessary to address the comment made by my esteemed friend and colleague Thanh Tran, general manager of our Houston operations. In his blog entry earlier this month, Thanh mentioned “I work with a bunch of geeks. But that’s ok, because I’m a geek myself.”

I feel that this might have been a light-hearted jab at my previous entry, which I will readily admit was steeped in a healthy dose of science geekery.

Now, Thanh says that he too is a “geek.” For the record, I would like to point out that keeping a MS Paint-drawn Buzz Lightyear image file archived on a hard drive somewhere for nearly ten years pretty much seals the deal on one’s level of geekishness.

Thanh, I don’t think you had to tell us you are a geek; consider it self-evidenced. Just kidding around, of course …

All kidding aside, I’ve been looking forward to this blog entry, because I wanted to take a moment to acquaint everyone with just a few of the many outstanding data center technicians that our customers have probably worked with in the past. The next time you are reading a support ticket, check the usernames of the employees working the issue with you, and see if you recognize any of these excellent technicians.

For example, Jac has been with The Planet for more than two years, and is currently a Level 2 data center technician. A few days ago I was reading a recent ticket that he was working on for a customer who was suffering from a bit of “new server syndrome.” That’s the critical window of time after someone orders a new server that we have all experienced at one time or another. The customer was hard at work attempting to get his configurations in place so the server could be usable for his business, but he was running into some pretty wicked permission errors that he simply could not deduce.

Jac responded to the customer’s questions quickly and effectively, adjusting the settings to address the issues. He then went above and beyond expectations, providing the customer with some very detailed documentation related to these permission configuration issues. The problem was wholly solved, and the customer was quite happy. But more importantly, the customer was enabled to address his server needs more quickly in the future as well. It’s all about responding to business needs precisely and rapidly, and Jac Lindley does an excellent job of helping our customers achieve just that.

Another Level 2 data center technician I wanted to “introduce” is Johnny L., who has been with The Planet for almost three years, and in that time he has proven to be a strong anchor of knowledge in our data centers. Johnny is a whiz when it comes to server hardware; I’ve seen him swap out bad drives and start degraded array rebuilds so quickly it would make your head spin.

The pace of work in our data centers can be dizzying at times, and seeing guys like Johnny work can be awe-inspiring. But in the midst of all this activity, the thing that makes a guy like Johnny stand out is his attention to detail. Anyone can whip through a dozen unique hardware upgrades in record time, but not many can do it with the level of accuracy and care that Johnny displays.

I’m proud of all the technicians we have working in the data centers, but I thought to would be great to introduce a couple of our techs to everyone. Keep an eye out for them in a support ticket coming to an Orbit portal near you!

- Britt

I was reading Thanh Tran’s blog post the other night and I realized that I had the other side of that story to tell. I joined Everyone’s Internet in 2001, as the original RAQ 4i servers were just starting to be deployed and most data centers were demanding a huge setup fee to go along with a monthly service charge of $600 or more. At the time, our core business was providing dial-up internet access and this “new hosting thing” was an experiment that the owners had decided to try out.

Our support started out with a focus on control panel guidance (who here remembers the Cobalt web interface?) and reboots (offered within 24 hours, when our sysadmin, Patrick Smith, would drive from the call center to the collocation facility and process a batch of requests). At the time, my primary job responsibility was to visit customer Web sites and make sure they did not contain any adult content, which they often did when you got beyond the home page. How’s that for a job description?

Things went along at a moderate pace until two things changed at roughly the same time. First, Sun purchased Cobalt and stopped playing nice, forcing us to expand our offering to white box dedicated servers. Second, we launched our first $1 setup fee sale. I remember that I had just taken over the Web Sales team, which was then part of our dial-up sales department.

All of a sudden our little six-person team was buried in inquiries, new orders and demands for upgrades. Heady time, especially when you consider how the rest of the “tech bubble” was in the middle of a serious meltdown, and all the economy pundits could talk about was the downturn in business and a growing recession.

Now that I think about it, the ride that started that day has never really ended. 1Ghz servers became 2.0GHz+; white box systems turned into all-Dell data centers; unmanaged dedicated hosting gave way to Custom Self-Managed Servers or Private Racks; and now EV1 has joined with The Planet.

What strikes me most about that last milestone is how we all thought we would be dealing with an alien race, and instead we found we were working with the same people we had always been working with. Sometimes literally, and yes I’m talking about you, Keith.

So here we are, just over a year later, and the company known for having the best data centers has joined forces with the company known for having the best network. We also have several new members of the family on board, determined to make sure we become known for having the best support ever. I only hope the next six years are exciting as the last six were. Then again, with a company like this, how could they be anything else?

- Aaron