Hosting companies often operate their data centers with The Wizard of Oz’s “Pay no attention to the man behind the curtain” mentality, so you never know what the data center operations look like on a day-to-day basis. You might say I’m one of The Planet’s “behind the curtain” employees … not only do I work in one of The Planet’s data centers, I’m also the overnight supervisor. While some of these activities may seem a bit mundane, each is designed to provide our customers with the very best operational environment. The processes and procedures documented here are performed by all shifts in all of our data centers.
Shift Change
Our overnight shift begins daily at 7:00 PM (CDT), with a hand-off of issues from the previous shift which may need special attention. These hand-offs are typically “work already in progress” issues like operating system reloads and a system upgrades.
Once this “shift change” is complete, and the new shift’s operations staff is in place, each member of the team has specific tasks to perform. These tasks include the assessment of scheduled maintenance during the shift, a review of pending customer orders and server upgrades, and response to reboot requests that were submitted during the shift change.
Following these initial checks, we move into normal operational status for a majority of the night. During this period, we monitor and respond to high priority issues (usually in the form of tickets) and undertake any scheduled maintenance work. In addition to this technical work, a staff member will also perform a perimeter patrol every four hours.
Perimeter Patrols
While the name doesn’t sound like much, the perimeter patrol is an integral part of our data center operations designed so our staff can constantly monitor and control the data center’s status and its operational readiness. Each patrol takes around an hour to perform and is a top-to-bottom inspection of our facility and server environment. To better understand the significance of the patrol, take a look at some of the system and facility checks.
- Temperature and Humidity Readings for all Environmental Units
Each phase of our data centers have as many as 18 heating, ventilation and cooling units. These units are operationally independent from the others and use a dual-compressor technology to provide heating, air conditioning, and humidity environmental control. Our data centers are kept at or below 76 degrees Fahrenheit, and if temperatures exceed that threshold on any one unit, redundant units are brought online and a facilities engineer is immediately contacted regardless of the time of day.
- Commercial UPS System Status Displays
Our data center is fully protected by redundant, commercial-grade uninterruptible power systems. In the unlikely event of an interruption to electricity supplied to our facility, these UPS systems provide stop-gap power and automatically signal our generators to start. Within seconds, our generators are running and providing power. In my data center, we’ve got 10 display panels to check for operational readiness on each patrol.
- Generator System Panels and Status Displays
There are two levels of generator monitoring in our data center. The first is a remote set of annunciation panels installed in our monitoring center. These displays provide an instant status of our power generating system. The second and more comprehensive set of screens we check are in our power generation and transfer control room. On those displays, we check important generator operational parameters such as engine coolant temperature, fuel level, and battery voltage. An interesting fact about our generators: the engines are temperature controlled, even when they are not running. If you ever win Jeopardy with that information, I’ll expect a cut of your prize.
- Electrical Power Transfer Switch Status Displays
Each one of our power transfer switches has a detailed display that shows us the status of the power entering our facility, whether backup power is online and available, and other, more detailed parameters. Here, we also check the total amperage in use by the facility for any abnormal variances.
- Life Safety System Status, Piping Pressures, and Fire Suppression Tank Levels
Our data center uses combined methods of fire suppression technology, so we regularly verify their operational status by reading air and water pressures, inspecting the piping for integrity and checking the master status panel in our monitoring center. Our fire suppression system is constantly monitored by our data center staff and an outside monitoring company. In the event of an alarm, emergency units are automatically dispatched to our facility.
- Exterior Doors and Intrusion Detection System Panels
All entry points are physically verified as being closed and locked on a regular basis. We have an intrusion alarm panel in our monitoring center which shows the status of our entire facility which is constantly monitored by data center staff.
- Closed Circuit Video Monitoring System
We employ a comprehensive video monitoring system in our data center with remotely controlled cameras mounted at strategic points both inside and outside of the data center.
- Outdoor Generator Enclosures and Engine Components
On each patrol, we actually open the generator compartments and peer inside at the massive engines which supply our data center with power during utility power outages to check the area around the generator for leaks, debris, and manually verify fuel level indicators. To get a sense of the size of the generator, check out Kevin’s “Data Centric” post.
- Outdoor Air Conditioning Condensers
For each of our heating, ventilation, and cooling systems inside the data center, there is a corresponding condenser outside the facility. We walk by each condenser and make sure the fans are running at optimal performance.
I’d love to sit and chat a little longer, but I’ve got some work to do. Assuming that this post is helpful and informative, I’ll start thinking of a few of the other tasks in the data centers that you might enjoy learning about.
-Eric
H2 Data Center Operations Supervisor (overnights)
Given the results of the poll we included in our previous blog post, I started looking for a good opportunity to meet the demand for more “Pictures from The Planet’s Offices.”
The year is winding down and the holidays are around the corner. As a nice reward for a year of hard work, the big bosses decided we’d have fantastic holiday parties in both Houston and Dallas welcoming our families, friends and business associates to The Planet. Thus The Planet Holiday Festival was born.
The Planet’s Network Operations Center, although located in a prominent spot here at our Houston headquarters, is shrouded in a bit of mystery. Not exactly a “whodunit”-type mystery, but enough that when I walk past the blue-lighted, mission control-type room during my daily jaunts around the office I wonder, “Why do the NOC technicians get to watch CNN all day?” (Hint: Not because of Anderson Cooper’s smoldering good looks.) “What’s the purpose of all those monitors at the front of the room, exactly? Can I pull up the same information on my desktop?” (Spoiler alert: I can!) And, “Why is the room blue?” 
