RSS

Pinboard Blog

A Cloud of My Own

Trigger warning: Server minutiæ, jargon, tiny little numbered footnotes

At the start of the year I decided move Pinboard from the rented servers it was running onto my own hardware. I've written before about the tradeoffs of various hosting strategies and my reasons for avoiding virtual servers, so I won't repeat my rationale here. But I'd like to stress that nothing about these posts is prescriptive. In fact, I have no idea what I'm doing. I do it, I write it up, and then wisdom pours down from the Internet.

So please don't do what I've done, unless it's the right thing for you.

Running my own hardware meant having to purhase three machines: a big server to run the site on, a slightly less big server for failover, and a wee little pizza box to distribute traffic between the two.

For the main server, I wanted lots of memory, solid state drives for the database, enough CPU so the machine could comfortably think its thoughts, and several terabytes of hard disk space to hold user archives.

Optimistic and foolhardy, I thought it would be fun to build this main machine from scratch - or rather, to have a long-suffering sysadmin pal physically assemble it for me and send it to me for configuration. A few hours on Newegg later and I had assembled a machine with 92 GB of RAM and about 8 terabytes of available storage:



Primary Server

Thing Component # Each Total
Barebone Server1 Supermicro Barebone 2U 1 $1,075
CPU 2 Intel Xeon E5620 2.40 GHz 2 $390 $780
RAM 8 GB sticks, EEC registered 12 $95 $1,140
Hard Disks 2TB Western Digital 7k 8 $200 $1,600
SSD 180 GB Corsair 2 $290 $580
SATA Card 3 LSI Logic 8-Port SATA+SAS 1 $235 $235
$5,410

Not long after, this machine arrived (beautifully assembled!) in a fifty pound Federal Express box:



And within a few minutes I had learned that setting up custom hardware requires a fair amount of expertise, and that trouble awaits the unwary novice.

Here's just one example: my first attempts to install Ubuntu failed because the computer would reboot midway through the install process. This seemed to happen about ten minutes after startup, even if I left the machine completely idle. At first I suspected overheating, but a look at the BIOS setup screen showed me the CPU was practically at room temperature.

After many, many hours of flailing, a kind soul instructed me to read about motherboard jumper settings. There I found the culprit. Like many servers, mine had a 'watchdog timer', a hardware circuit that reboots the computer if it is not regularly reassured by a background process that everything is okay. The feature is intended for data centers, where it is not always convenient or possible to have someone press a button on a frozen machine.

Turning this timer off required moving a tiny plastic cap on the motherboard one pin to the right.

This was pretty representative of the problems I faced. It was not obvious at what level things were going wrong — bad component, misconfigured hardware, misconfigured firmware, misconfigured OS, or faulty driver — and testing hypotheses was frustratingly slow because of how long it took the server to boot. Moreover, some problems (like a subtly broken Ubuntu driver for my SATA card) would not even show up until the machine had been running for several hours, making them very difficult to trace.

In other words, debugging this stuff was exactly like debugging software, except that I had no experience to draw on and no gut feeling for what might be causing a given problem.

Particularly frightening was the ease with the machine could turn to molasses if I mis-set a motherboard switch, arranged memory cards in the incorrect order, or used the wrong parts. The systems on the motherboard were capable of running at a wide range of speeds, and they would cheerfully and silently slow down to accomodate the slowest component.

These pitfalls were well-documented in the manual, but navigating them required Talmudic levels of attention to the text and left me feeling paranoid and alone.

While certainly motivating, spending hours trying to unbrick a five thousand dollar server was not a joyous journey of discovery that I wanted to repeat. Even though I had had expert help in physically assembling the machine, getting it running had still consumed days of my time. I decided that building these things from scratch was best left to people who knew what they were doing, or had time to spare.

So for my remaining servers, I looked to a local company called Kingstar, recommended to me by Luke Crawford. They proved to be a very pleasant mom-and-pop operation that built me two custom servers for less than it would have cost to purchase the components individually. These arrived at my door (delivered by my sales rep in person!) completely assembled and tested.

Server number two - the failover machine - was very similar to its companion. It has a little less memory, and a lot less storage (since I would not need the capacity for a while, I wanted to wait for hard drives prices to drop a little bit):

Failover Server

Thing Component # Each Total
Basic Server 2U Servon XD1210 1 $2,285
CPU (Intel Xeon E5606) 2
RAM 4 GB sticks, EEC registered 18
Hard Disks 400 GB Western Digital 7k 2 $70 $140
Hard Disks 2TB Seagate 5k 2 $250 $500
Hard Disks 2TB Seagate 7K refurbished 2 $220 $440
$3,365

The final server was a slim 1u machine intended to serve as a load balancer and front-end cache, so I could switch the site between the two big machines without having to change DNS settings. I threw a pair of SSDs into it to make sure it could boot quickly, and to give me more caching options.

Load Balancer

Thing Component # Each Total
Server 1U Servon XS1316 1 $1,075
CPU Intel Xeon X3430 quad core
RAM 8 GB sticks, EEC registered 4
SSD 120 GB Intel 2 $200 $400
$1475


The grand total for the three machines was $10,250, so if you joined Pinboard in the second half of December, that's where your money went. Amortized over thirty-six months, this comes to a monthly cost of $286, not including colocation fees. But that's a topic for my next post.


Notes:

1: A barebones server is like the Hamburger Helper of rack servers. It's a pre-assembled and tested combination of motherboard, enclosure, and power supply - you add memory, CPU and storage to taste. Supermicro makes a variety of really terrific servers like this in every possible size.

2: The last time I built hardware [2001], choosing a CPU involved two steps:

  1. Intel or AMD?
  2. How many GHz can I afford?

The advent of multi-core processors has changed everything, and at this point Intel and AMD offer about six thousand subtly different variants each. Fortunately, Newegg was bundling this particular processor with the enclosure and I took this path of least resistance.

3: I have no idea what this thing does. Four black wires enter, two red wires leave. My sysadmin pal told me I would need this to make the SSDs visible to the rest of the system, and so in it went.

—maciej on May 31, 2012



Pinboard is a bookmarking site and personal archive with an emphasis on speed over socializing.

This is the Pinboard developer blog, where I announce features and share news.




How To Reach Help

Send bug reports to bugs@pinboard.in

Talk to me on Twitter

Post to the discussion group at pinboard-dev

Or find me on IRC: #pinboard at freenode.net