I was recently reminded of an old friend from India who moved to California a year back or so for a new gig as the networking guy at a mid-sized company. For simplicity’s sake, let’s just call him Raj (note: names have been changed to protect the innocent). As are most people starting a new job, he was eager to make a strong first impression by doing all the standard stuff really well, and he wanted to be responsive to any end-user request, especially anything coming from his new boss. On his first day, he was invited to a senior staff meeting and took detailed notes of all the stuff that was going on. Raj wanted to figure out how he could help — from a networking perspective. Now my friend is not like Apu from The Simpsons or anything, but English is his second language.

He heard exec after exec talk about how they didn’t have enough “bandwidth” to finish some project or another, and if they just had more “bandwidth” they’d be so far ahead of the game. Raj decided this was his opportunity to spring into action. He dutifully began analyzing RTG charts and even started installing network response testing agents. He definitely found some bottlenecks with some congested segments. Raj began building a plan to move from a number of shared segments to switched fabric to the edge for some of these apparent power users that needed more “bandwidth.”

He took the proposal to his boss, who reviewed it and complimented him on a thorough job and well articulated argument. Raj’s boss then proceeded to calmly and very politely explain to him what the execs meant when they referred to “bandwidth.” His boss was just happy that Raj didn’t have signing authority for that much gear. Needless to say, Raj now calls me a fair bit to make sure he’s got English vernacular down.

It just struck me as funny because the story came up as we were launching our unmetered bandwidth by Cogent. We were going through the same discussions to figure out how much capacity we needed relative to demand from our customers. There has also been conversations about which of our multiple data centers we’d provision to accommodate growth. English is my second language, and let’s just say I chose my words carefully after talking with Raj.

- Urvish

Last month Domain Keys Identified Mail (DKIM) was accepted as a chartered IETF effort, and since then it’s been touted as the next big thing in anti-spam. In the interest of clearing up some personal confusion about the framework, I went straight to the source and dove headfirst into RFC4871 and DKIM.org to get a good look at it.

If you’re not familiar with DKIM already, it’s a mechanism that allows Mail Transfer Agents (MTAs) and Mail User Agents (MUAs) to cryptographically sign outgoing e-mails in a manner that allows the receiving mail agents to verify the signer and the message integrity. It provides this mechanism through the use of a public/private keys and a key server (currently implemented using DNS).

A quick glance at the top of the RFC shows that the DKIM standard is backed by some pretty important industry names: Sendmail, Inc., PGP Corporation, Yahoo! Inc., and Cisco Systems, Inc. Wide-spread industry support is extremely important when adopting a new standard, especially those relating to anti-spam tech. So I also took time to scan the list of organizations on dkim.org that publicly support DKIM. Some of the more significant names include: AOL, EarthLink, EBay/PayPal and qmail.org.

After determining the kind of support DKIM has already received, I delved into the RFC abstract and noticed a fairly important message that seems almost slipped in at the end:

"Protection of email identity may assist in the global control of "spam" and "phishing." (RFC4871) 

Read that again: “… may assist in the global control of …” In other words, DKIM is not specifically an anti-spam technology. In fact, if you read the “DKIM: Introduction and Overview” article on DKIM.org, you’ll see it specifically stated on the “Overview of DKIM” slide. Instead, it provides a mechanism for attempting to verify that the message sender is really who they say they are.

When a message is signed with a DKIM signature, the mail agents add the signature to the header of the message. The signature contains the domain for the sender; a selector to use in the public key lookup; the cryptography algorithm used to generate the digest; the digest itself; a public-key lookup mechanism (currently only DNS is supported); and a number of configuration parameters.

On the other end of the pipeline, the receiving agent uses the signature fields to create a DNS query that allows the message header and contents to be verified against the sending domain’s public key. Public keys are looked up via domain name, and it can be assumed that the owner of the domain is responsible for their DNS records, hence there’s no need for a centralized authority to maintain and verify the public keys.

I’ve grossly over-simplified the mechanism in my description, but I don’t want to go too deep into the actual mechanics of DKIM (that’s what RFCs are for). I do want to point out a few things about using the standard in an anti-spam capacity.

First of all, as previously stated, DKIM is not an anti-spam mechanism itself. There is no reason a spammer could not simply setup DKIM for their domains, sign their messages, and have the receiving mail agents happily report that the message is validly signed.

Of course, a user could always just block any messages received from the signing domain as soon as it’s determined to be spam, but what if the senders start using arbitrary domain names to send a single mass mailing? After all, “asdfghjkl.com” can be signed just as easily as “iamaspammer.com.” Once a single mailing is sent out on asdfghjkl.com, the spammer could discard the domain and move on to asdfghjkm.com and so on.

The eventual goal is to have all legitimate mail signed. At that point, any unsigned messages could be dropped as “bad,” forcing the spammers to use DKIM as well. This goal is well into the future, but for now DKIM suggests that unsigned messages simply fall-through to regular spam filters for processing.

However, this goal means that a lot of DKIM’s worth is wrapped into the idea of senders being able to build a reputation. Signers who frequently send “good” or “bad” messages will build a reputation for doing so. Spam filters can then be configured to be more lenient on DKIM signers who have “good” reputations and more strict on those who have “bad” reputations.

Here-in rests the problem … once all mail is being signed, how do we determine which senders are sending “good” messages and which are sending “bad” messages? Certainly the messages may still be passed through regular spam filtering mechanisms, but what happens if the message makes it through them? How do we utilize DKIM as a reputation system? The only way I can think of is to provide the end-user receiving the message a way to report it as “bad.”

This brings us back to the “spam” button at the top of your e-mail console – along with a new host of questions. For example, should the reputation system be a centralized global system, or should ISPs setup a system for use by only their users? Does every message that’s reported as spam degrade the signer’s reputation? How do MUAs and MTAs query the reputation system? There are a lot of open-ended questions regarding what to do after everyone (including the spammers) is properly signing their messages.

In conclusion, as a standard for signing and verifying messages, DKIM certainly fits the bill, but there are too many open questions relating to its implementation as an anti-spam tool after it’s widely implemented as a verification tool. These questions need to be answered before DKIM really can gain traction as an anti-spam heuristic.

Maybe I’ll write that RFC …

- Tekkie

Resources: DKIM home page - RFC4871

Interested in purchasing new servers to keep up with increased demand? The Planet recently launched two new server specials that deliver valuable savings on the latest technology and server solutions.

Take advantage of the Let FREEdom Ring special, with any new server order eligible for free setup, a free RapidSSL^® Certificate and free 2GB DiskSync backup.

For servers with a base monthly price of $150 or more, the special also includes free control panel software from a wide selection of industry leading vendors, including cPanel, Plesk, Ensim and Helm.

Sign-up soon. This offer is available on all new server orders from June 11 – July 6, 2007.*

Are your servers spread across a number of hosting providers? Send us the competitor’s invoice and The Planet will beat current server pricing by 10 percent. Running from June 11 – July 31, 2007, the Lowest Hosting Price on The Planet also delivers free setup and one month free to migrate the server(s).*

If The Planet is unable to beat the competition’s pricing on an equivalent server configuration, customers will be provided with a one-year RapidSSL^® Certificate.

Stop by again soon for the latest deals from The Planet.

- Katie

A couple of weeks ago I finished reading a book called Pushing Ice (by Alistair Reynolds). It’s a hardcore sci-fi novel that follows a team of comet miners as they’re asked to push beyond their normal mission to explore strange happenings with one of the moons of Saturn, and how their work affects the galaxy. One of the more intriguing plotlines is how no matter what obstacles they face, the group is able to fall back on their motto - “We push ice, it’s what we do.”

As you may have seen, we’ve recently announced that we now have in excess of 100GB of transit Internet capacity. This is a staggering amount of bandwidth, and thinking back to just 4 or 5 years ago I never really envisioned that we would need that kind of capacity.

But these days I talk to vendors in terms of 10G ports (it was 1G ports as late as a year ago), and I’m now starting to look at the 100G standards that are being developed and trying to figure out when I’ll be able to evaluate and deploy them into our network.

Internet bandwidth growth has been on a nearly exponential growth curve for years, and as our connections get larger the threats we have to deal with get larger as well. We’ve seen DDOS attacks that exceed 10Gbps in the last few months, and other attacks that have been as large as 6-8Mpps. Attacks of this scale would have crippled backbone networks just a few years ago, but these days they simply raise eyebrows.

I’ve read many articles about the approaching Internet crunch - and how the Internet is just a few steps away from a massive implosion. These kind of articles seem to pop up every year or two - and every time I see them I chuckle. It reminds me of the doomsday predictions of Y2K - and what a non-event that was. The simple fact is that contrary to what some of these reports seem to indicate, there are a lot of smart people out there working on solving some of the problems before they manifest.

This is not to say that there aren’t issues out there. I recently wrote about ARIN’s declaration regarding IPv4 space exhaustion, and the need for providers to start looking at moving to IPv6. This issue poses some serious problems for all Internet users. Perhaps it’s my inherent trust in man’s ability to overcome any issues that he encounters through his ingenuity but I don’t see this as a doomsday scenario but rather as a great challenge that we can and will deal with and overcome.

This brings be back to my opening. As network engineers we’re constantly faced with daunting issues related to scale and traffic growth. The Internet routing table is now over 240,000 routes (it was less than 40,000 just 10 years ago) and the bigger that routing table gets, the more we have to squeeze out of our routers to accommodate the growth. No matter what dire predictions are made, or obstacles we encounter out there, we’ll just keep pushing packets, because that’s what we do.

Time is something we all take for granted. It is the only finite measurement that all businesses and individuals rely on. We have doctor appointments, meetings to attend, bills to mail and deadlines to meet. What happens when you can’t depend on time? How do businesses and individuals operate without a universal standard of time? In truth, operating without time is probably one of the worst possible situations to deal with and, trust me, it does happen!

While doing my tour through technical support, I found that most server time clocks were incorrect. This led to a number of issues ranging from an SSL not working, inabilities to log-in and difficulty tracing a problem through log files. I don’t blame the administrators of those servers, because it all comes down to how our clocks tick.

There are two clocks with every computer, the hardware clock and the system clock. CMOS uses the hardware clock to keep track of time running at 64Hz (usually). The system clock runs at whatever the kernel is compiled to run at. In Linux, the default 2.6 kernel line defaults to 1000Hz** for increased interactivity at the cost of more overhead. But, you can recompile between 100Hz, 250Hz and 300Hz. Each set of tick rates has their advantage and shortcomings. Soon to come, there will be a tickless Linux kernel, making the problem even more interesting. Let’s save that for another day.

As the CMOS ticks away at 64Hz and the kernel at 1000Hz, there is a drift from one clock to another called a skew. As the clocks drift apart the system clock “runs” faster than the CMOS clock, and the time between the two no longer serves as a concrete form of measurement. Instead, you have a CMOS clock that is probably running a bit slow, and a system clock that is running quite a bit faster. How do you use the clock as a reference clock for anything at this point?

Enter Network Time Protocol. It keeps all the clocks in the world within milliseconds of each other (when configured to use NTP). Administrators will setup systems to sync against logistically close Tier 2 NTP servers.

I won’t write a how-to on the setup of NTP servers, there are a number of articles that already exist out on our glorious series of tubes. I will explain a little how NTP does work, and how it will apply to every server you deal with going forward. NTP and its corresponding tools will adjust your system’s clock to accurately reflect your designated offset from UTC time. NTP always responds in UTC time, but the NTP tools running on your server will make the time zone adjustment.

If your clock is fast, it will slow down the conversion of jiffles/ticks until your system clock more accurately reflects the correct time. If your clock is slow, NTP will automatically adjust your time. Time is a linear progression and does not deal well with “future” dates. Many tools utilize modification times of their configuration files to see if they need to be reloaded. IDS monitoring tools utilize modification time, checksums and log times. Payment gateways are sensitive to time as well. Having a date set in the future breaks many logic gates, and can leave you troubleshooting issues for hours or destroy precious financial records.

At The Planet, we have a time server for you to use to help offload some stress from public time servers: ntp.theplanet.com

Feel free to utilize this on your servers, if it isn’t already setup for it. Don’t forget to set-up a cron job to sync your hardware clock against your system clock. I highly suggest reading this article from Vmware on the subject as ticks and drift get very complicated when you add virtualization.

Good luck. I hoped I cleared things up a bit!

- Kevin Landreth, RHCE

** Most recently changed to 250Hz, but it really depends on who compiled the kernel

According to Wikipedia.org, O’Reilly Media coined the phrase Web 2.0 sometime in 2004. The term has since become ubiquitous, but it’s difficult to pin down when Web 2.0 — the focus of the Internet community on user-generated content — actually began.

When social-networking sites like MySpace began making news and gaining popularity of course I had to take a look. What I found was largely underwhelming. Not to disparage the creators of MySpace — that community is a force of nature, and something to be respected and admired by all Internet entrepreneurs — it just didn’t seem like a completely new concept.

After all, forums are comprised of users who generate their own content, upload photos to their profiles and create online communities, usually based on a particular interest or hobby. These are communities nonetheless. Who knew sites like Web Hosting Talk and our own humble customer forums would be the unacknowledged godfathers to an entire movement called Web 2.0?

Hopefully the popularity of forums won’t diminish like other user-generated content formulas. I’ve always felt the advantage forums have over other types of communities is their ability to foster dialogue in an arena overcrowded with monologues. And yes I am aware of the irony of using a blog to make that statement.

It’s my belief that both mediums of communication are essential to building relationships with our customers. In fact, I’d like to think that we add value that even goes beyond exchanging credit card numbers for dedicated servers. Blogs provide an opportunity to communicate a concept, idea or opinion to its fullest, while communication in our forums is mostly reactive versus informative.

So until next time … I’ll see you on the forums!

- Brooke

When the subject of writing a gaming related blog for The Planet came up, I was a bit apprehensive. I’m new to blogging and felt rather noobish. The whole thing is reminiscent of a creative writing assignment, except I’m graded by a whole community of people.

After much deliberation, I decided that my entry into the blogosphere would focus on the lessons learned at my second job. Aside from my position as a product line manager and my role with the Insomnia365 gaming server line, I was recently promoted to CEO of a small company with approximately 120 employees.

The previous CEO left the company, and the search for a new leader commenced. For those that remained, the responsibility of rebuilding deterred any immediate volunteers. I thought about the employees that remained, and what would happen if things were left to flounder. I decided to speak up. The remaining management staff nominated and passed a referendum naming me the new CEO of the company.

I admit, I accepted the position thinking that I knew what was in store. But I didn’t expect the wild ride the last month has been. My first week on the job was very disheartening. A number of long time “employees” took long vacations or sick days, and the general morale of the company was at an all-time low.

My main priority focused on building employee confidence in the new direction of the company. Most of the HR-related problems were tackled with one-on-one sessions to address individual concerns. I worked through a number of issues, and now understand what a guidance counselor feels like.

As one of my first acts as CEO, I hand-picked two new company officers from the ranks to help rebuild the “office.” The new officers had a strong history in the “field” and were very good at teambuilding.

We needed heavy hitters and the new management staff worked around- the-clock checking applications and doing interviews with potential new employees. Advertisement for positions were posted on all of the major “job” sites, and what started as a trickle grew to a river of potential new recruits.

The addition of fresh new talent helped rejuvenate the team, and we were back on track the following week. Training was surprisingly easy, having established policies and procedures in the past.

Currently, the company has two main teams running projects simultaneously and our “company” growth has sharply increased. The resource shortage was overcome, and company morale has been restored.

This second job has taught me a number of important lessons, but the biggest lesson learned is that people have different personal goals and reasons for being with a particular company. But, no matter the reason, communication is the best way to resolve any problem. Lack of communication and being out of touch with your customer base is a recipe for failure.

I’ve also learned that success breeds success. I’m taking many of the lessons from this second job and continue to apply them here at The Planet.

As I finish up my first blog entry, I’m also getting ready to put on my other hat and log-on to my second job. Being the guild master of a World of Warcraft raiding guild is hard work, and I have to get my core team of raiders through the next dungeon tonight!

It was tough rebuilding the guild when the old guild master left, taking many of the core players with him. But we’ve managed to recruit and overcome the dungeon bosses we were stuck on. Isn’t it amazing the parallels you can draw from a virtual world … this is a new age we live in.

P.S. If you have a level 70 Warlock on the Illidan server, we have a spot open!

- Sassan