The Planet Blog

With Web2.0 upon us, some customers have difficulty deciding when the time is right to make a new server purchase, particularly amidst a sea of buzz words like AJAX, Ruby, and Content Delivery Network. Which server is best for you? Should you get the Celeron or the Dual Clovertown? How can you choose between them without being bitten by over- or under-purchasing? How many times have you purchased a server with your boss looking over your shoulder? Hopefully, I can share a little of my professional experience to make your next server-buying experience a little easier … maybe when your boss is hounding you about how much you are spending on your next server, you can point to this definitive guide from someone who has done a few tours of duty in the industry.

What’s Your Server’s Application?

It’s a simple and straightforward question: What is your server’s application? I’m not talking about whether you are using apache, fastcgi, php, python, lighttpd, IIS6/7, asp or .NET. I am interested in how your server is configured and its purpose: Is it a Combo (http+db), a Workhorse (http or appserver), a File Server (http, ftp, rsync), or a Media Server (http site + files)?

While I boiled down the general “web servers” category into only those few types, I can almost guarantee that your web server will fall into one of them, and each type has its own needs.

The Combo (http+db)

While this is one of the most inefficient configurations, it happens to be the most common. Combo boxes rely heavily on CPU and RAM because they not only process the web request, but also handle the database queries. Databases queries eat up RAM and so do most web-scripting languages. The server’s CPU and hard disk access will probably be the most limiting factors in this type of application. I wouldn’t recommend spending a lot of money on RAID configurations or extra disks in a Combo machine since you would be better suited to run two purpose-customized non-RAID machines — with money to spare on backup space — for about the same price.

When you are able to break your Combo server down into separate dedicated http and database machines, you will be able to better tune and customize each server’s configuration for most efficient disk patterns, VM subsystem pressure and RAM usage. It may sound a little confusing, but I’ll explain a bit more about the different types of servers’ needs in my next post.

The WorkHorse (http or appserver)

When you don’t have a database on your http or appserver, you are looking at more of a workhorse-style server application. This configuration typically needs a focus on RAM and CPU without much demand for disk access. Typically, you will see servers like this bundled with Tomcat or ColdFusion. Most PHP applications suggest this setup because the ever-growing intensity of database queries and the amount of post processing in the scripting languages can put a strain on a box serving both the http and database requests. As a note, when you’ve set up your dedicated http or appserver box, you’ll have to decide whether to use mysql_connect() or mysql_pconnect() for your web application. A word to the wise: pconnect() is great if used in the right circumstances so don’t write it off if you don’t know how to use it or because it crashed your server last summer while you were on vacation. *wink*

The File Server (http, ftp, rsync)

File servers are the easiest beasts to set up. RAID and RAM!! RAM is for the file cache (which prevents direct disk-read requests), and RAID configurations accommodate multiple spindle heads. You will need 4 or 5 disks to make up for the RAID5 “write penalty.” I’d advise you not to get duped by reading about RAID1’s performance in reading from multiple disks. While it may be “smart” by definition, most hardware RAID controllers do not run “smart” RAID1. Don’t neglect the CPU on these toys since CPU plays an important role in bus transfers between the RAM, your disk controller and network connections. Dual processor or dual core machines would be ideal here.

The Media Server

These are the same as file servers but need a lot more CPU power and probably a separate database server (for the same reason explained in the “Workhorse” explanation). More spindles here = more better.

Do I Need to Upgrade?

Don’t rush into a server upgrade “just because” … look at your current utilization patterns — install sysstat so you can at least use SAR and make RTG’s, and estimate your usage. Is your server running at 25% or 75%? If you are at 75%, you might need to split your web and database servers. If you are at 75% and already have made that split, you may need a much better box with application-level caching and threads. I’d recommend against over-shooting your needs since the monthly cost can eat away at you in the end. If you are speculating a 10-15% increase in usage, just double up on the most important server resources go increase them by about one-and-a-half times if you already have a beast of a machine.

Don’t get caught up with RAID configurations if you would be better served with two machines with more RAM and CPU. In the end, it’s not about uptime or how much you are paying for a server each month; it’s about the most efficient way to use what you have so you can spend your money where it is needed.

Making the Move

Most of the forum posts I’ve read were about people who hate migrating from server to server or about how horrible the process is for their customers typically don’t have much experience in the process. The more you do it, the better you become at it (not really a novel idea, I know). My experienced advice: you need at least two weeks to get it done and done right. Why two weeks? Everyone knows DNS is a pain and so are custom-installed perl/pear modules … remember in ’04 when you installed Compress::LZO for that one customer?

I hope this helps!

-Kevin Landreth, RHCE
Technology Architect

Certifications have been an industry standard for years as a barometer to gauge an individual’s depth of technical expertise. I would disagree that a certification can serve as a barometer of technical expertise. After working in this industry for more than four years and obtaining only one certification, I’d have to say my perspective has changed from certifications being a de facto standard to more of a gauging utility on a resume.

For example, if I were to say that I designed three separate automation processes for deploying Linux and third-party software, then one would wonder if I really had accomplished such a task. I mean, Al Gore did invent the Internet, right? But, if you saw I had an RHCE[i] and a Java enterprise architect certificate[ii] then my certifications may lead you to believe that it’s possible that I did accomplish such a task.

Now if I had designed a completely scalable N+1 tier mirroring repository service without any certifications, then this may prove that certifications offer nothing more than an industry recognition that I know how to do my job. But, in designing the repository service, I’ve already proven my value to the company based on the solutions I’ve delivered.

Here’s where I’m torn:
Should I get more certifications? And if so, which ones?

The company thinks I’m valuable already — I’m blogging! — so I’m not gunning for more money here. What I am looking for is more industry recognition, but does that come through networking with peers and vendor relationships or does that come more from light networking but having a nice slew of certifications to say that I know what I’m talking about here. This has been frustrating me to some degree as I am well-rounded[iii] in the industry and the corresponding technologies, but I don’t know if the cost/benefit ratio is right for either approach.

I guess the next question that always follows is did I finish school, and the answer is “no.” I probably should have, and maybe one day I will. I guess if technology degrees weren’t so vendor-specific I would have finished. I did switch to logistics but then I got serious about my career and didn’t go back.

Of course, 50+ hour weeks and 16 credit hours can do that to a person.

Oh well. Such is life. The question remains: to cert or not to cert. I welcome your thoughts and feedback.

- Kevin Landreth, RHCE, Sr. Systems Architect @ The Planet

[i] RHCE # 804007071224202
[ii] Do they even have these? Either way I don’t have one….
[iii] While I have gained weight since working here, that’s not what I’m talking about!

Time is something we all take for granted. It is the only finite measurement that all businesses and individuals rely on. We have doctor appointments, meetings to attend, bills to mail and deadlines to meet. What happens when you can’t depend on time? How do businesses and individuals operate without a universal standard of time? In truth, operating without time is probably one of the worst possible situations to deal with and, trust me, it does happen!

While doing my tour through technical support, I found that most server time clocks were incorrect. This led to a number of issues ranging from an SSL not working, inabilities to log-in and difficulty tracing a problem through log files. I don’t blame the administrators of those servers, because it all comes down to how our clocks tick.

There are two clocks with every computer, the hardware clock and the system clock. CMOS uses the hardware clock to keep track of time running at 64Hz (usually). The system clock runs at whatever the kernel is compiled to run at. In Linux, the default 2.6 kernel line defaults to 1000Hz** for increased interactivity at the cost of more overhead. But, you can recompile between 100Hz, 250Hz and 300Hz. Each set of tick rates has their advantage and shortcomings. Soon to come, there will be a tickless Linux kernel, making the problem even more interesting. Let’s save that for another day.

As the CMOS ticks away at 64Hz and the kernel at 1000Hz, there is a drift from one clock to another called a skew. As the clocks drift apart the system clock “runs” faster than the CMOS clock, and the time between the two no longer serves as a concrete form of measurement. Instead, you have a CMOS clock that is probably running a bit slow, and a system clock that is running quite a bit faster. How do you use the clock as a reference clock for anything at this point?

Enter Network Time Protocol. It keeps all the clocks in the world within milliseconds of each other (when configured to use NTP). Administrators will setup systems to sync against logistically close Tier 2 NTP servers.

I won’t write a how-to on the setup of NTP servers, there are a number of articles that already exist out on our glorious series of tubes. I will explain a little how NTP does work, and how it will apply to every server you deal with going forward. NTP and its corresponding tools will adjust your system’s clock to accurately reflect your designated offset from UTC time. NTP always responds in UTC time, but the NTP tools running on your server will make the time zone adjustment.

If your clock is fast, it will slow down the conversion of jiffles/ticks until your system clock more accurately reflects the correct time. If your clock is slow, NTP will automatically adjust your time. Time is a linear progression and does not deal well with “future” dates. Many tools utilize modification times of their configuration files to see if they need to be reloaded. IDS monitoring tools utilize modification time, checksums and log times. Payment gateways are sensitive to time as well. Having a date set in the future breaks many logic gates, and can leave you troubleshooting issues for hours or destroy precious financial records.

At The Planet, we have a time server for you to use to help offload some stress from public time servers: ntp.theplanet.com

Feel free to utilize this on your servers, if it isn’t already setup for it. Don’t forget to set-up a cron job to sync your hardware clock against your system clock. I highly suggest reading this article from Vmware on the subject as ticks and drift get very complicated when you add virtualization.

Good luck. I hoped I cleared things up a bit!

- Kevin Landreth, RHCE

** Most recently changed to 250Hz, but it really depends on who compiled the kernel