RSS

Pinboard Blog

Going Colo

Last time around, I wrote about buying hardware. This post is about finding a place to put it.

But I have to start with a kind of meta-cautionary note:

When you run an active website by yourself, you will come across many opportunities to geek out. This stuff is fascinating at every level of abstraction. But the same curiosity and desire for mastery that helped you succeed as a programmer will now lure you into burning massive amounts of time if you allow yourself to dick around. Every new problem domain will attempt to seduce you with its beautiful, chess-like subtlety. You have to ignore this and respond by playing checkers.

Colocation is a case in point. There are wheels within wheels! So many factors to consider! But ultimately it's about getting machines racked with the blinking lights pointing forward.


That said, on to the details.

If you're not familiar with it, a colo is a data center that takes your money in return for a safe place to put your computer. Apart from physical security, the facility typically provides you with climate control, power, bandwidth, a range of IP addresses, and often some kind of 'remote hands' service if you need to have hardware installed or buttons pressed.

The basic unit of space is a metal enclosure called a rack; it looks like a reinforced gym locker. Racks are divided horizontally into 42 notional slices about the thickness of a pizza box, called rack units (abbreviated U).

All server hardware is made to fit these racks. A 1U server is plenty powerful for most needs, but if you start throwing in hard drives, the enclosures get thicker. For reference, my fairly modest servers are 2U high and can hold ten terabytes of fanfic.

If you are bloody-minded enough, it's perfectly possible to have a shelf installed in your rack and stand regular desktop-style computers on top of it. There are reasons to avoid this (mostly to do with desktop stuff not being designed for such conditions), but not everybody is deterred.

You pay a monthly fee based on some combination of space, bandwidth, and power.

Until you rent multiple racks, you remain a minnow in the eyes of any datacenter. A quarter rack (10U) is about the smallest chunk you can rent directly. Some resellers will rent individual rack units, but these plans are typically quite restrictive and may not include physical access to your stuff.

Bandwidth is more complicated. Here are three common ways to bill it:

  • Pay for a set amount of terabytes transferred each month, like a phone data plan.
  • Pay for a given connection speed (10 Mbps or 100 Mbps or 1 Gbps).
  • Use what you like and get billed at the 95th percentile.

The third option is nice if you can get clever about how you use the connection, but it's also easy to incur huge bills if you mess it up. Remember, checkers!

It's worth knowing what the other end of that network cable plugs into. Some datacenters are connected to multiple upstream providers, others hang suspended from the Internet by a single fiber. The datacenter will usually boast about its incredible connectivity on its website if it has it.

If you're coming from the world of dedicated servers like me, this may be the first time you have to think about power. A colo will typically charge you for circuit capacity, not actual use (unless you are a big enough customer to get a metered plan). Often you will pay by the ampere. Note that you are not allowed to use the full rated capacity of a circuit; if you get 20 A only 16 of those amps are usable.

Unfortunately, figuring out your power needs is not as simple as reading numbers off the back of your server. The best thing to do is buy a Kill A Watt and run the machine for a while at a high-normal workload. In my case, I've found roughly 1 ampere = 1 rack unit.

I had a short list of requirements when I started looking for colocation:

  1. A quarter rack of space.
  2. 100 Mbps dedicated. I wanted to avoid having to monitor and throttle my own Internet use. A capped, dedicated link meant one less thing to configure.
  3. Round-the-clock physical access to my stuff, so I could fix and tinker as necessary.
  4. Somewhere reachable by car within a few hours.
  5. Low seismic risk.

Unforunately, it's not possible to get #2, #4 and #5 at the same time in California. The only place where data centers will not plummet into the sea when the Big One hits is Sacramento, and bandwidth prices there are extremely high, on the order of $20/Mbps.

So I decided to eat it, seismically speaking. I would try to mitigate the risk by hosting in two locations, each within driving distance from home. I reasoned that the risk of an earthquake big enough to affect two data centers was negligible compared to the risk of general equipment failure, so it was better to host somewhere shaky but reachable.

The first place I selected was Hurricane Electric, in Fremont. They came highly recommended by a friend who had done business with them for years, and I liked their transparent pricing. They offered me 8U at 2 amps for $200/month with a 100 Mbps unmetered data plan. While two amps was only enough electricity for one server, it was all I needed.

The newer Fremont data center is a gleaming white building with a $0.25 soda machine, very pleasant staff, and vast halls of available space. You get 24/7 access to the facility, hassle-free operation, and terrific technical support via email. And the building sits pretty much directly atop the Hayward fault.

I bought my second chunk of rackspace through Luke Crawford of prgrm.com, who has recently started building a colocation empire.

I had dealt with Luke in early Pinboard days, when I rented one of his extremely inexpensive virtual servers. While virtual hosting proved to be a poor fit for an I/O-intensive site like Pinboard, I came away impressed with his integrity, expertise, and exemplary transparency about outages and issues. Many months later, even though I was no longer a customer, Luke took the time to advise to me at length about hardware choices when I was getting ready to buy my own equipment.

So I jumped at the chance to do business with him again. For $206, I got 10U, 4 amps, and 100 Mbps connectivity in a San Jose data center about 14 miles from Fremont. Best of all, the facility was not built directly on top of an earthquake fault!

Instead, it rested on a bed of alluvial sand that will liquefy and drain north into the Bay the moment the last trumpet sounds.

Overall, pretty typical for California.

My total colocation fees come to $406/month. Adding in the amortized cost of hardware brings the bill to $780, which compares favorably with what I had been paying for leased servers.

It would certainly be possible to host the site for a few hundred dollars less, but this where that bit about chess and checkers comes in again. I now have two custom, redundant machines at different spots on the power grid. If this stuff all goes up in flames, it should at least make for a good story.

Some additional data from seismically sane providers I talked to:

Raging Wire* in Sacramento offered me a half-rack for $500, but told me 100 Mbps would cost an additional $1600. Apparently this is quite typical of Sacramento bandwidth prices. From the point of view of the Internet backbone, Sacramento is a remote and hostile planet.

Fiberhub in Las Vegas is offering 10u, 5A, 100 Mbps for $250/month + $150 setup. This is the nearest earthquake-free colo I could find in the three figures. But the idea of heading to Vegas every time a hard drive sickened seemed to introduce its own, very different kind of risk.


* What is it with these aggro facility names? Rather than Hurricane Electric or Raging Wire, I would much prefer to host with "Calm Multihomed Oasis" or "Granite Mountain" or " Cooling Breezes Pure Sine Wave Mount Bitmore".

—maciej on June 22, 2012



Pinboard is a bookmarking site and personal archive with an emphasis on speed over socializing.

This is the Pinboard developer blog, where I announce features and share news.




How To Reach Help

Send bug reports to bugs@pinboard.in

Talk to me on Twitter

Post to the discussion group at pinboard-dev

Or find me on IRC: #pinboard at freenode.net