Archive for the ‘Tech Stuff’ Category

The Toll Grows Louder

Thursday, August 16th, 2007

Will CharnockARIN Issues Statements About Future Allocation Policies - SEO to blame?

A few weeks ago I blogged about ARIN’s IPv6 notice. Interestingly — and I don’t take credit for this, by the way –ARIN has now decided to attempt to proactively address the threat of an evolving secondary market for IP space. As IPv4 addresses are a limited resource, it only makes sense that individuals and groups will try to capitalize on the coming number crunch as an opportunity to create a black or grey market for unused IP space. ARIN has issued a statement (read here) that reinforces their intention to adhere to the spirit of the original policies and basically lays down the gauntlet for other registries to do the same. The paragraph that best sums it up for me is:

The purpose of this memorandum is to assure the community that the democratic principles of Internet governance will be adhered to by ARIN, the Regional Internet Registry serving Canada, many Caribbean and North Atlantic islands, and the United States.(7) The resource-allocation policy under which ARIN operates has been produced through an open, transparent, and democratic process over more than a decade. ARIN is fully dedicated to preserving universal access and stable functionality of the Internet, and our policies do not encourage profit-driven speculation in the Internet addresses.

Because the system relies on a certain amount of self-governance, we have to rely on the people running these registries and their willingness to resist market temptations and continue to enforce the current policies that are set forth. We also have to rely on the honesty of the people requesting IP allocations from the registries.

As the situation tightens, it will be interesting to see how the various registries respond. Ultimately, the market will demand that we shift away from the IPv4 space altogether and into the IPv6 space. At The Planet, part of our strategic planning is centered around being ready for this shift.

We see more and more requests for IP space for non-traditional reasons — SEO being a primary culprit. There is a lot of confusion out there about what variables are used in assigning rankings on the various search engines, and it appears to be driving more and more people to grab as much IP space as possible to try and improve their rankings. Higher ranking = more $$$$.

If the search engine companies really care about the Internet and its usability, it would behoove them to issue a statement clarifying the importance — or lack thereof — of IP addresses in the ranking of search engines. So that we have a definitive answer on whether or not it really matters. Until then, my fear is that the problem is only going to get worse, and we’ll be running out of IP addresses sooner rather than later.

In the meantime, I would encourage end users to really research the topic themselves before asking for that extra IP allocation. You might find out that you’re paying for something you don’t really need.

- Will

142857

Thursday, August 2nd, 2007

Anthony LedesmaIf you follow the link, you’ll see that this number is unique in so many ways.

I won’t go into the specifics of the number, but will provide a general overview along with why I’d be proud to have anybody associate this number with The Planet and myself.

Continue multiplying with further integers and you can add the first number to the remaining 6 numbers to continue this cyclic permutation, or end up with the same nice high repeating number. What is a cyclic permutation? And why would one be proud to be associated with such a sequence? I’ll let you dive further into the details. What we have here is a number that repeats itself in cycles (cyclic permutation) when multiplied by integers.

1 × 142,857 = 142,857

2 × 142,857 = 285,714

3 × 142,857 = 428,571

4 × 142,857 = 571,428

5 × 142,857 = 714,285

6 × 142,857 = 857,142

7 × 142,857 = 999,999

What I see here is repetition at its finest. Repetition in a company is essential whether it be in support, accounting, information technology, networking, marketing or any other department. Repetition ensures that you end up with the same, or at least similar, results. The results should either be those that are expected or close to, with the ability to correct them. We can now associate the expected results to 142857 and the unexpected results to 1142856. When we end up with the figure 1142856, we know what needs to be done in order to correct it: 1 + 142856 = 142857.

In IT departments we should always follow this rule. While every project may be different, and require some fine tuning, we should get used to doing our job the same way every time. This ensures that if someone is sick or on vacation, any other team member can simply jump in and continue the work without breaking stride. Whether someone is evaluating new network equipment or a new control panel everything must be documented from start-to-end.

In our tech support, it’s a rule we work to follow so that we provide consistent service to our customers.

Let us know what you expect and what you were provided with support. This will help us to keep up with your ever-changing world, while we provide the infrastructure and support to ensure your business growth.

- Anthony

The Bandwidth Confusion

Wednesday, July 11th, 2007

Paul DaigleThis is my first blog post, so I thought I’d share a funny story about my first few days at The Planet. Coming from a predominantly networking background in the Internet service provider (ISP) world, I was thrown for a loop the first time I looked at our server descriptions.

As I browsed through, I noticed the labels for “bandwidth” and “uplink port speed” as two separate items. As I said, coming from the ISP side of the technology industry we defined “bandwidth” in a completely different fashion than the hosting industry. To me, bandwidth has always been defined as how big the pipe is that will transmit or carry data. So when I saw the “Bandwidth” and “Uplink Port Speed” labels, it really threw me for a loop (nothing like a routing loop to really screw up your day – HA!).

In the ISP world and more to the actual definition, bandwidth has meant the width or depth of allocated bands of frequencies in a transmission channel. It’s the width of the spectrum a signal occupies. Think of it in terms of tubes or pipes – a two-inch pipe or tube has less width or depth to carry or transmit things than say an 18-inch pipe or tube. In this example, the two-inch pipe would be an ADSL line vs. a DS-3 line.

“Bandwidth” always seems to be confused with data rate or capacity – otherwise known as how much “stuff” can I send at a given speed (usually measured in a per second time unit). This would be closely related to what we label as “uplink port speed” and “bandwidth,” respectively and as we define it, more than the definition of “bandwidth” as I have technically defined it.

So that brings me to the next thing that may come across your mind – what do “bandwidth” and “uplink port speed” mean with The Planet and the hosting industry?

Let’s hit “bandwidth” first. Most, if not all, of our listings show them in some thousands of GB (GigaBytes). Our Conroe’s come with a default of 2500 GB of “bandwidth” per month. From our definition, that means you can transfer 2500 GigaBytes of data in one month – it’s an aggregate of both inbound and outbound data to and from the Conroe server. But still, what does that mean?

We’ll, let’s say that you had a database that was 2500GB in size (massive database!). If you started transferring it from this server to say another server, outside of The Planet, and you wanted to do it over 30 days (approximately, one month), then using a nice little conversion formula (2500GB/month * 1 month/30days * 1 day/24 hours * 1 hour/60 minutes * 1 minute/60 seconds * 8 bits/1 Byte) we see that the minimum “uplink port speed” should be a 7.72 Mbps connection – a slower than 10Mbps link connection, which was the old de-facto standard that we used to use for connecting PCs and servers to LANs. But this gives you an idea of how your decision in “uplink port speed” can be directly proportional to how quickly you need to do business.

Now we all know that we have a faster connection than 7.72Mbps, especially when the Conroe is defaulted with a 100Mbps “uplink port speed.” So what does this mean when we run the formula this way? Let’s say that we use all of the “pipe” allocated to the 100Mbps “uplink port speed” for the entire month (to remain consistent with the above example). Plugging our numbers into the formula, but in reverse (100Mb/second * 1 Byte/8 bits * 60 seconds/1 minute * 60 minutes/1 hour * 24 hours/1 day * 30 days/1 month) we see that we can transmit inbound and outbound about 32,400GB or 32.4TB per month of data! WAY beyond the specified 2500GB or 2.5TB per month that we default to the Conroe server.

They are related, but do not necessarily equal one another in relation to their monthly ability. What’s important to see is that you know what both are capable of. One tells you how fast you can transmit data given the bandwidth (this is the “uplink port speed”) while the other caps or limits the amount of data that can be transmitted in a given month to and from that server (“bandwidth”).

This month, The Planet has launched its Cogent “unmetered bandwidth” offering and this is where customers can quickly learn how valuable even 10Mbps of “unmetered bandwidth” can be! A 10Mbps, unmetered “uplink port speed” would yield 3,240GB or 3.24TB of “bandwidth” per month! A very nice savings at $200/month!

- Paul

Anatomy of a Ticket

Tuesday, July 10th, 2007

Anthony LedesmaI’ve spent many years in various tech support roles throughout my career, regularly reviewing issues customers face. At The Planet, our goal is to provide excellent service for our customers, which usually comes through our support tickets. Most of our customers use the ticket system, so I thought I’d offer some thoughts on how to ensure we help you resolve outstanding issues quickly and thoroughly.

Here is an example of a support ticket that provides just about every detail that’s necessary to remediate the issue:

—–SNIP—–

Department: Support

Subject: New York City AT&T DSL unable to connect to my server in DLLSTX6

Server: servername-1.2.3.4

Body: My customer, Jon Smith, is unable to ssh into 1.2.3.4 on tcp/22 from 255.255.255.255. We have insured that Jon is not being blocked via iptables and he can connect to Apache on tcp/80.

[root@server ~]# iptables –nLChain INPUT (policy ACCEPT)target prot opt source

destination Chain FORWARD (policy ACCEPT)target prot opt source

destination Chain OUTPUT (policy ACCEPT)target prot opt source destination[root@server ~]#

[root@server ~]# traceroute 255.255.255.255

myserversgateway (1.2.3.1) 0.004 ms 0.020 ms 0.011 ms

1 dsr01.dllstx2.theplanet.com (12.96.160.9) 0.901 ms 0.944 ms 1.174 ms

2 …3 ……10 …11 255.255.255.255 (255.255.255.255) 42.211 ms 50.421 ms 45.114 ms[root@server ~]# ping 255.255.255.255PING 255.255.255.255 (255.255.255.255) 56(84) bytes of data.

64 bytes from 255.255.255.255 (255.255.255.255): icmp_seq=0 ttl=43 time=49.0 ms

64 bytes from 255.255.255.255 (255.255.255.255): icmp_seq=1 ttl=43 time=45.4 ms

64 bytes from 255.255.255.255 (255.255.255.255): icmp_seq=2 ttl=43 time=45.5 ms

64 bytes from 255.255.255.255 (255.255.255.255): icmp_seq=3 ttl=43 time=54.0 ms

64 bytes from 255.255.255.255 (255.255.255.255): icmp_seq=4 ttl=43 time=46.1 ms

— 255.255.255.255 ping statistics —

5 packets transmitted, 5 received, 0% packet loss, time 4003ms

rtt min/avg/max/mdev = 45.453/48.060/54.077/3.277 ms, pipe 2

[root@server ~]#

[customer@remoteserver ~]$ telnet 1.2.3.4 80

Trying 1.2.3.4…

Connected to myserver.tld (1.2.3.4).

Escape character is ‘^]’.

HEAD / HTTP/1.0

HTTP/1.1 404 Not Found

Date: Fri, 06 Jul 2007 17:34:37 GMT

Server: Apache/2.0.52 (Red Hat)

Connection: closeContent-

Type: text/html; charset=iso-8859-1

Connection closed by foreign host.[customer@remoteserver ~]$

ssh 1.2.3.4ssh_exchange_identification:

Connection closed by remote host[customer@remoteserver ~]$ telnet 1.2.3.4 22

Trying 1.2.3.4…

Connected to myserver.tld (1.2.3.4).Escape character is ‘^]’.

Connection closed by foreign host.[customer@remoteserver ~]$

You can login to my server with the credentials listed in Orbit. You can reach me any time at (555) 555-5555.

Thank you,Bob

Customerbob@remote.tld

—END SNIP—

Without ever logging into the server, our trained technicians are able to immediately identify the problem. The issue in this case is that either through a script(bfd) or other means, the remote user is being denied by TCP WRAPPERS (man 5 hosts_access). We are able to see that iptables are not blocking any connections, and we are able to reach the server without issue on another TCP port that’s not typically controlled by the wrapper. The customer ensures that the ticket subject was descriptive so that our technicians are able to evaluate the ticket in the support system queue. Each technician typically has a different area of expertise, so we work to steer the ticket to the right person for the job. Here is a list of subjects that provide helpful information for the support techs responding to the ticket:

  1. cPanel: WHM is not responding. Server is accessible via SSH
  2. Apache: My .cgi is giving a 501 error
  3. cPanel: Apache is segfaulting after I upgraded PHP
  4. Exim: User so-and-so cannot send outgoing form mail after tweaking security in cPanel
  5. Server kernel panics daily. Need to have the RAM tested/replaced
  6. MySQL: forums refusing more than 100 concurrent users
  7. Reboot Request**
  8. Network: San Jose, CA – 350ms RTT via Sprintlink

** #7 should not be a Support Ticket but a Reboot Ticket.

At The Planet, tech support is always eager to help resolve issues quickly and efficiently, so when customers add a bit of detail in the subject line, we’re able to ensure the ticket is routed to the most appropriate individual.

Just a few tips to help bring a quick resolution to a number of support issues. Thanks for stopping by.

- Anthony

cPanel Tips

Friday, July 6th, 2007

Jason MathewsHello, world! I’m Jason Mathews, the dayshift supervisor for technical support. I’ve been playing with computers since 300 baud and in the industry since 1992. I’ve been a network engineer, systems administrator and all around go-to guy for anything powered by electrons. So I have a lot of personal experience helping customers deal with issues when they arise.

Managing dedicated servers and the associated infrastructure can be pretty complex, and we’re here to help with that.

As an example, cPanel has gone to version 11, and there have been some support requests since the upgrade. This is no slight on cPanel — I’m not picking on them. It’s typical to see some initial discomfort when new versions come out, and usually there are some easy fixes. Since we’ve heard from some customers about this recent version, I thought it might be helpful to provide some tips in our blog to help:

1) Find your current version of Perl. You can do that by typing in:

perl –v

at the command line. Make sure that your servers have Perl 5.8.7 at a minimum. 5.8.8 would be better, and to get the installer package for that, run this in SSH asroot:

wget http://layer2.cpanel.net/perl588installer.tar.gz

tar zxvf perl588installer.tar.gz

cd perl588installer

perl install

This will get perl up-to-date and recompile a whole lot of perl modules that cPanel depends on. cPanel isn’t updating perl directly, and if it’s anything below 5.8.7, many modules won’t work, recompile at all or will provide strange output.

2) Run this script and check the output: /scripts/updateuserdomains

This will read the user files in /var/cpanel/users and make any needed synchronization with other system files, such as Apache and ftp.

Sometimes, however, the user configuration will have conflicts you may not have noticed, mostly by one account having control of a domain while that domain is still active on another account. This inconsistency can cause a lot of trouble later on, so you’ll want to edit the user files directly, remove the offending domain, then run the script again.

3) Verify all of cPanel is updated to version 11.

perl -c /scripts/wwwacct

If this command doesn’t have any errors, then everything should be at the proper version. If you get errors, something might be missing, so run this:

/usr/local/cpanel/bin/checkperlmodules

/scripts/upcp -force

This will double check all perl modules and run the update in force mode, overwriting existing cPanel functions with the latest ones. If this still doesn’t work, let us know, and we can try more detailed fixes, or we can escalate the issue to cPanel directly and they can provide a proper fix.

4) Ensure that you are using the maildir format in email. The previous versions of cPanel used both mbox and maildir formats, but mbox is being phased out, and maildir is much more efficient anyway. To check this,run:

/scripts/convert2maildir

This will tell you what your current mail format is, and offer you the chance to convert to maildir and backup current mailboxes before the change. This script is pretty reliable, I haven’t run across any problems with it. It will install the latest courier-imap to handle the new format. This will also mean that Horde and Squirrelmail will then work with the new format, and Neomail will go away, since that has long since been deprecated.

That’s pretty much it. These are the steps we would follow in looking at any recent issue with cPanel. I would welcome any comments or suggestions for the next posting. If there’s something you want to know about technical support and what it is we do, I’ll write about it.

- Jason

Domain Keys Identified Mail – Clearing Up the Confusion

Tuesday, June 26th, 2007

Andrew StarodubLast month Domain Keys Identified Mail (DKIM) was accepted as a chartered IETF effort, and since then it’s been touted as the next big thing in anti-spam. In the interest of clearing up some personal confusion about the framework, I went straight to the source and dove headfirst into RFC4871 and DKIM.org to get a good look at it.

If you’re not familiar with DKIM already, it’s a mechanism that allows Mail Transfer Agents (MTAs) and Mail User Agents (MUAs) to cryptographically sign outgoing e-mails in a manner that allows the receiving mail agents to verify the signer and the message integrity. It provides this mechanism through the use of a public/private keys and a key server (currently implemented using DNS).

A quick glance at the top of the RFC shows that the DKIM standard is backed by some pretty important industry names: Sendmail, Inc., PGP Corporation, Yahoo! Inc., and Cisco Systems, Inc. Wide-spread industry support is extremely important when adopting a new standard, especially those relating to anti-spam tech. So I also took time to scan the list of organizations on dkim.org that publicly support DKIM. Some of the more significant names include: AOL, EarthLink, EBay/PayPal and qmail.org.

After determining the kind of support DKIM has already received, I delved into the RFC abstract and noticed a fairly important message that seems almost slipped in at the end:

"Protection of email identity may assist in the global control of "spam" and "phishing." (RFC4871) 

Read that again: “… may assist in the global control of …” In other words, DKIM is not specifically an anti-spam technology. In fact, if you read the “DKIM: Introduction and Overview” article on DKIM.org, you’ll see it specifically stated on the “Overview of DKIM” slide. Instead, it provides a mechanism for attempting to verify that the message sender is really who they say they are.

When a message is signed with a DKIM signature, the mail agents add the signature to the header of the message. The signature contains the domain for the sender; a selector to use in the public key lookup; the cryptography algorithm used to generate the digest; the digest itself; a public-key lookup mechanism (currently only DNS is supported); and a number of configuration parameters.

On the other end of the pipeline, the receiving agent uses the signature fields to create a DNS query that allows the message header and contents to be verified against the sending domain’s public key. Public keys are looked up via domain name, and it can be assumed that the owner of the domain is responsible for their DNS records, hence there’s no need for a centralized authority to maintain and verify the public keys.

I’ve grossly over-simplified the mechanism in my description, but I don’t want to go too deep into the actual mechanics of DKIM (that’s what RFCs are for). I do want to point out a few things about using the standard in an anti-spam capacity.

First of all, as previously stated, DKIM is not an anti-spam mechanism itself. There is no reason a spammer could not simply setup DKIM for their domains, sign their messages, and have the receiving mail agents happily report that the message is validly signed.

Of course, a user could always just block any messages received from the signing domain as soon as it’s determined to be spam, but what if the senders start using arbitrary domain names to send a single mass mailing? After all, “asdfghjkl.com” can be signed just as easily as “iamaspammer.com.” Once a single mailing is sent out on asdfghjkl.com, the spammer could discard the domain and move on to asdfghjkm.com and so on.

The eventual goal is to have all legitimate mail signed. At that point, any unsigned messages could be dropped as “bad,” forcing the spammers to use DKIM as well. This goal is well into the future, but for now DKIM suggests that unsigned messages simply fall-through to regular spam filters for processing.

However, this goal means that a lot of DKIM’s worth is wrapped into the idea of senders being able to build a reputation. Signers who frequently send “good” or “bad” messages will build a reputation for doing so. Spam filters can then be configured to be more lenient on DKIM signers who have “good” reputations and more strict on those who have “bad” reputations.

Here-in rests the problem … once all mail is being signed, how do we determine which senders are sending “good” messages and which are sending “bad” messages? Certainly the messages may still be passed through regular spam filtering mechanisms, but what happens if the message makes it through them? How do we utilize DKIM as a reputation system? The only way I can think of is to provide the end-user receiving the message a way to report it as “bad.”

This brings us back to the “spam” button at the top of your e-mail console – along with a new host of questions. For example, should the reputation system be a centralized global system, or should ISPs setup a system for use by only their users? Does every message that’s reported as spam degrade the signer’s reputation? How do MUAs and MTAs query the reputation system? There are a lot of open-ended questions regarding what to do after everyone (including the spammers) is properly signing their messages.

In conclusion, as a standard for signing and verifying messages, DKIM certainly fits the bill, but there are too many open questions relating to its implementation as an anti-spam tool after it’s widely implemented as a verification tool. These questions need to be answered before DKIM really can gain traction as an anti-spam heuristic.

Maybe I’ll write that RFC …

- Tekkie

Resources: DKIM home page - RFC4871

Pushing Packets

Wednesday, June 20th, 2007

Will CharnockA couple of weeks ago I finished reading a book called Pushing Ice (by Alistair Reynolds). It’s a hardcore sci-fi novel that follows a team of comet miners as they’re asked to push beyond their normal mission to explore strange happenings with one of the moons of Saturn, and how their work affects the galaxy. One of the more intriguing plotlines is how no matter what obstacles they face, the group is able to fall back on their motto - “We push ice, it’s what we do.”

As you may have seen, we’ve recently announced that we now have in excess of 100GB of transit Internet capacity. This is a staggering amount of bandwidth, and thinking back to just 4 or 5 years ago I never really envisioned that we would need that kind of capacity.

But these days I talk to vendors in terms of 10G ports (it was 1G ports as late as a year ago), and I’m now starting to look at the 100G standards that are being developed and trying to figure out when I’ll be able to evaluate and deploy them into our network.

Internet bandwidth growth has been on a nearly exponential growth curve for years, and as our connections get larger the threats we have to deal with get larger as well. We’ve seen DDOS attacks that exceed 10Gbps in the last few months, and other attacks that have been as large as 6-8Mpps. Attacks of this scale would have crippled backbone networks just a few years ago, but these days they simply raise eyebrows.

I’ve read many articles about the approaching Internet crunch - and how the Internet is just a few steps away from a massive implosion. These kind of articles seem to pop up every year or two - and every time I see them I chuckle. It reminds me of the doomsday predictions of Y2K - and what a non-event that was. The simple fact is that contrary to what some of these reports seem to indicate, there are a lot of smart people out there working on solving some of the problems before they manifest.

This is not to say that there aren’t issues out there. I recently wrote about ARIN’s declaration regarding IPv4 space exhaustion, and the need for providers to start looking at moving to IPv6. This issue poses some serious problems for all Internet users. Perhaps it’s my inherent trust in man’s ability to overcome any issues that he encounters through his ingenuity but I don’t see this as a doomsday scenario but rather as a great challenge that we can and will deal with and overcome.

This brings be back to my opening. As network engineers we’re constantly faced with daunting issues related to scale and traffic growth. The Internet routing table is now over 240,000 routes (it was less than 40,000 just 10 years ago) and the bigger that routing table gets, the more we have to squeeze out of our routers to accommodate the growth. No matter what dire predictions are made, or obstacles we encounter out there, we’ll just keep pushing packets, because that’s what we do.