Sunsetting Delicious

August 21, 2014

Since the birth of the site, Pinboard has always offered a Delicious sync feature. You could enter your Delicious username on the settings page, and the site would periodically poll your public Delicious feed and add any new bookmarks it found.

Because Delicious appears to be in a terminal coma, and because working around bugs in their RSS feed has historically consumed a lot of development time, I am going to turn Delicious sync off effective October 1, 2014. If you want to keep hooking the services together, you will still be able to do it through an outside service like IFTTT or Zapier.

Of course you will still be able to import all your Delicious bookmarks, as well as export Pinboard bookmarks in a format that Delicious can (sometimes) read. The only thing going away is the automatic sync.

—maciej on August 21, 2014

Researching Link Rot

August 09, 2014

This week I'll be running a little experiment in link rot, in preparation for an upcoming conference talk. I'm interested in quantifying long it takes for a typical link to go offline, and if this rate is steady or changing with time. Pinboard now has enough bookmarks (about 100 million) to make this information interesting.

This research is important because we don't have a lot of data for how link rot affects stuff people actually care about. Presumably things that you've bookmarked are more important to you than some random URL off the street.

To run the experiment, I am going to be drawing a few thousand links at random from the entire pool of Pinboard bookmarks. This will include private bookmarks, which make up about half the Pinboard collection. I'll use a combination of scripts and my own weary hands to figure out what proportion of links still point to the original material saved. The URLs I look at will not be associated with your username, and no one except me will look at them.

I will publish some aggregate information about what I find, and use it to seek glory, and persuade people to sign up for archiving. But I won't release anything that could lead back to specific users or links.

If you are uncomfortable with this research and wish to opt out, please email me with your username, and I'll keep your bookmarks out of the pool. If you have questions, ask me on Twitter or email me privately, and I'll be happy to answer them.

As a reminder, the site's privacy policy is here.

Thank you!

—maciej on August 09, 2014

Pinboard Turns Five

July 09, 2014

Today marks five years since I launched the website that my mom still refers to as 'the other bedbugs'. Happy business birthday to me!

Any site that aspires to be an archive starts life with a credibility problem. The Internet is strewn with the corpses (or in some cases, zombies) of sites that once promised to save your links forever. As people keep discovering, building a bookmarking site is easy, but making a business of bookmarking is hard. Like one of those leathery, spiny plants that is able to thrive in the desert where everything else dies, I have tried to find ways to adapt to this hostile business environment. And I have feasted on the flesh of my rivals!

I raise this brimming skull to the awesome group of users and fellow-travelers who have made it possible.

It's my tradition to post updated statistics about the site:

The biggest surprise (to me) is how predictable Pinboard has been over the past three years. Users come and go, like on every site, but the number of active users stays roughly the same. And the site makes roughly the same amount of money (around $200K) every year.

If you've ever run a small website, you'll recognize how weird this is. Typically everything in a small project—traffic, user count, revenue—is spiky. You spend a long time treading water and then big events happen that dominate everything else. This was true for the first two years I ran the site, but since then, things have settled down remarkably.

I regret that I totally forgot to keep downtime stats this year. There wasn't a lot of it, but I should probably track it better so I can brag about it next year, unless it goes up, in which case I will never mention it again.

Now back to some beard-stroking:

I see my role much like a small-town praire banker in the 1880's. My job is to project an aura of calm, solvency, and permanence in an industry where none of those adjectives applies. People are justifiably risk-averse when it comes to their bookmarks, and they are looking for stability. This means several things at once:

On the most basic level, the site just has to work.

On the design level, it means not futzing with stuff unnecessarily, except for bug fixes and basic improvements. Luckily there is so much work to do on Pinboard that I am immune to the temptations of a redesign. If there is a feature (or bug) you love in 2014, chances are excellent it will still be there, like a cherished friend, years from now when your trembling and aged hands go to make that final click.

Finally, there is stability on the business level. This means persuading people (including myself) that I am going to stick around, and then actually earning enough money to do that.

The money part turns out to be easy. People will pay for a decent service. As long as you stay small and don't forget to have revenue, you too can build a bookmarking website. There is plenty of room to specialize!

My strategy of pre-emptively antagonizing anyone who might possibly have an interest in acquiring or funding the site has worked wonderfully. In five years, I haven't received a single email from an investor or potential acquirer. The closest I came was a few months ago, when the new Delicious owners reached out to me about providing "vision", but I think they were just unfamiliar with my oeuvre. They learned quickly.

So the biggest risk in a project like this remains burnout.

Avoiding burnout is difficult to write about, because the basic premise is obnoxious. Burnout is a rich man's game. Rice farmers don't get burned out and spend long afternoons thinking about whether to switch to sorghum. Most people don't have the luxury of thinking about their lives in those terms. But at the rarefied socioeconomic heights of computerland, it's true that if you run a popular project by yourself for a long time, there's a high risk that it will wear you out.

It's not the fact of working on just one project that's the problem. This dude, for example, has spent much of his life building a Boeing 777 out of manila folders. Another guy (always dudes!) is slowly excavating his basement with toy trucks.

What burns you out is the constant strain of being responsible for a lot of other people's stuff.

The good news is, as you get older, you gain perspective. Perspective helps alleviate burnout.

The bad news is, you gain perspective by having incredibly shitty things happen to you and the people you love. Nature has made it so that perspective is only delivered in bulk quantities. A railcar of perspective arrives and dumps itself on your lawn when all you needed was a microgram. This is a grossly inefficient aspect of the human condition, but I'm sure bright minds in Silicon Valley are working on a fix.

Perspective does not make you immune to burnout. It just makes burnout less scary. I've gone through a few episodes since starting Pinboard, and I'm sure there will be more to come. People have been very understanding about my occasional need to flee the Internet. I find that the longer I run the site, the more resistant I become to the idea of ever giving it up, even if I need to take the occasional break. It is pleasant to work on something that people draw benefit from. It is especially pleasant to work on something lasting. And I enjoy the looking-glass aspect of our industry, where running a mildly profitable small business makes me a crazy maverick not afraid to break all the rules.

Most of all, I'm gratified that people have been patient and considerate over the whole lifetime of the project. There has been a lot of goodwill sent my way that makes my job vastly easier. Thank you to all the people who have used the site over the years, and the many people who have helped me build it and keep it running. To my competitors: I will crush you! To everyone else: you're wonderful! Upgrade!

—maciej on July 09, 2014

Cloudy Snake Oil

April 19, 2014

I have nothing against Amazon S3, but I was astonished today to find this claim on their FAQ page:

Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities.

This is an impressive number, but it's utterly dishonest to make such claims. It implies that there is a less than one-in-one-hundred-billion chance that Amazon will abruptly go out of business, or that a rogue employee will cause massive data loss, or an unexpected bug will result in massive data loss, or a defect in storage media will cause millions of devices to fail silently, or a large solar flare will destroy equipment across three data centers, or that a comet impact will destory three data centers, or that a nuclear exchange will destroy three data centers.

I think these events are all incredibly unlikely, but none of them is one-in-a-hundred-billion unlikely. Yet here is Amazon not only making that argument, but implying that you can safely use S3, a service that launched in 2006, for another ten million years.

Rare events are rare! That's why promises past five or six nines of reliability are functionally meaningless. At that point the "unknown unknowns" must overwhelm any certainty you have about what you think your system is doing.

The risks you failed to model will become obvious in retrospect, and make for an entertaining post-mortem, but that won't get anybody's data back.

Promises like Amazon's should serve as a kind of anti-marketing, suggesting that the company has not thought seriously about the limits of risk assessment and planning.

I suggest the following rule of thumb: if you can't count the number of nines in the reliability claim at a glance, it's specious.

Of course this rant is available in book form, phrased better than I have here. But it's worth repeating at every opportunity.

—maciej on April 19, 2014

Heartbleed and Pinboard

April 09, 2014

By now you may have heard about the heartbleed bug, which affected many websites that use encryption. This is the Spanish Flu of security bugs—it hit almost everyone and took a disproportionate toll on the healthiest, those sites that followed security best practices.

Servers affected by the bug (including the Pinboard site and API) could be tricked into sending private information that happened to be in memory. This included authentication cookies, passwords, secret API tokens, and any data you posted to the site. There is also evidence that the bug could expose a site's private key, which would mean anybody eavesdropping on a Pinboard connection could decipher it. The bug was live from the spring of 2012 until I patched the servers on Monday night.

Worst of all, there's no way to tell from logs if anyone's data was exposed. It's possible that no one looked at any Pinboard data; it's also possible that the site was completely compromised.

This morning, I issued a new TLS certificate for the site, with a new private key. Now that the servers are trustworthy, please do these two things:

1. Change your Pinboard password. Go to https://pinboard.in/settings/password and have at it.

2. Reset your API token. On that same page (https://pinboard.in/settings/password), click the reset button. You'll need to update any outside services and apps that use the API token to authenticate. (Remember never to share your Pinboard password with any third party, no matter how nicely they ask. Outside sites should be able to get all the access they need using only the API token.)

In layman's terms, the bug was the equivalent of asking a stranger "hey, what's up?" and having them tell you their most private thoughts, going on about their divorce, sharing their credit card info, whatever was on their mind at the time. You could keep asking "what's up" as often as you wanted, and hear new things each time. Worst of all, the stranger would have no recollection that it had happened.

Of course, I heard about heartbleed before it was cool. The servers were patched by around 7 PM on Monday night, California time, before half the Internet started casually playing with Python scripts that exposed the bug.

So only truly malicious people could have seen your Pinboard secrets. Hooray!

In awful times like these, it's good to stop and reflect on the timeless wisdom of the Pinboard security page:

"Please do not store truly sensitive information in your Pinboard account."

I don't want anyone getting shot because I used the wrong Linux distro.

This is terrible! Good luck out there! Please feel free to email me if you have questions, or concerns, or would just like to kvetch.

—maciej on April 09, 2014

Planned Maintenance March 22

March 19, 2014

The site will be down for maintenance this Saturday, March 22 from 19:00-21:00 California time (UTC/GMT -7). For people in Berlin, that's 04:00 Sunday. For Australians and kiwis, that's Sunday afternoon. The database is just about to run out of storage space, and unfortunately there's no way to free up room without shutting the beast down for a few minutes. I'll do my best to keep the downtime brief, and will post updates to Twitter as I go.

—maciej on March 19, 2014