Alphabetical Sorting

People have been asking for alphabetical sort order for two years, and I have stubbornly ignored them since it didn't seem particularly useful to me. Such are the hazards of having a site dictator! However, there has been such consistent demand for the feature that I've finally added it for the user+tag pages, and naturally I've immediately found myself wondering how I did without it.

I'll roll out alphabetical sort to other parts of the site shortly. Wherever you see a small arrow and 'date - title' selector near the navigation links you can click to change sort order. This is 'sticky' for that page type - once you've asked to sort by title on a user tag page, you'll keep seeing that sort order until you change it back.

—maciej on October 18, 2011

Spikes, Logs and the POWER CLAW

Before launching Pinboard I created the usual ritual series of spreadsheets to try to anticipate traffic, data storage requirements, and revenue given a variety of scenarios. And like everybody else I have found these spreadsheets to bear little relationship to reality.

The problem with trying to model this stuff is that we find ourselves in a domain where a small number of rare events can completely dominate the data. Here's an all-time graph of new Pinboard users per day, for example:

This chart shows user signups over time, but you'd see the same graph for every metric of interest - traffic, Twitter mentions, cups of coffee consumed by the developer. Just seven days account for half of all Pinboard revenue.

The mechanics of this are fmailiar. Someone writes an article about you, or you're featured on a Top 10 list, or a meteor hits your competitor and very briefly the full attention of bored people on the Internet is yours.

Somewhat counterintuitively, while the timing and size of the events is unpredictable, the overall pattern is regular. Here's what the same graph looks like if you rank all the days by number of new users and plot them on a log-log scale:

This kind of plot is the hallmark of the POWER CLAW. If your stats look like this, you can know with some confidence that your day-to-day experience will not prepare you for a few extraordinary days that will matter most for your project.

You can also expect to spend most of your time grinding away, waiting for those extraordinary days to arrive. Rare events are rare - it says so right there in the name!

Of course, many other people have noticed this phenomenon, and there's a terrific book devoted entirely to it. But there's something about human psychology that makes it very hard to internalize the idea. I'm still making the damned spreadsheets.

If you run a web thing, please consider sharing your own log-log plot with the world, obfuscating whatever you need to to feel comfortable. I'm very curious about how many young sites see a similar traffic pattern, and whether it diminishes as you grow.

---

* Back around 2003 the blog world became fascinated with power laws and exponential distributions due to a Clay Shirky essay. POWER CLAW has been my mental shorthand for this kind of beard-stroking ever since.

—maciej on October 16, 2011

Time Zone Support

You can now tell Pinboard what time zone you're in on the settings page, and the display date for bookmarks will be adjusted accordingly.

Just a reminder that you can replace the default "2 days ago" style creation date with an exact date and time by setting the 'Show exact date and time on bookmarks' preference. Even if you don't turn the preference on, you can always see the exact time any bookmark was saved by mousing over the saved date text.

—maciej on October 12, 2011

Four Recent Changes and Two Useful Things

1. A new feature called 'tag subscriptions', which allows you to follow any number of global tags in one place. I had earlier assumed that people who wanted to do this would just build a folder for themselves in an RSS reader, but several users argued persuasively that it was much more convenient to have this within Pinboard.

   To turn the feature on, go to your settings page. You'll see a 'subs' link appear at the top of your user page as well as in the tag cloud, and you'll see a 'subscribe' link on every global tag page. You can also just type in tags on the subsription page directly. The page will show you all public bookmarks that contain at least one of the tags you've subscribed to - think of it as a recent page filtered by tag.

2. New visual highlighting on all global tag, network and recent pages to show if you already have a URL in your own bookmarks. These bookmarks will now appear in light grey, with a 'has_bmark' class (for those of you rewriting CSS at home). The intention is to make it faster to skim these pages for new links.

3. Things have moved around on the 'settings' page. There's now a dedicated privacy tab where you can control things that affect your visibility to others. Over the coming weeks I'll be adding some user discovery features to the site, so I wanted to put all privacy-affecting options in one place. Of course everything will default to the most paranoid setting (Pinboard's no Facebook) and I'll write about the new features here as they're released.

4. I've removed the downloads feature altogether. For the unfamiliar, this allowed you to grab the last 25 bookmarks from any page in a format suitable for local viewing. I still think this is a really neat feature, but I've removed it because it has a lot of moving parts and requires maintenance out of proportion with its usefulness.

Now for the two useful things - if you're looking for old friends from your Delicious network, check out this wonderful network name mapper or this vast Google spreadsheet.

—maciej on October 11, 2011

The Fans Are All Right

I've had a couple of emails and tweets asking somewhat cautiously why the popular page has filled with slash fiction. That's because the fans are coming!

I learned a lot about fandom couple of years ago in conversations with my friend Britta, who was working at the time as community manager for Delicious. She taught me that fans were among the heaviest users of the bookmarking site, and had constructed an edifice of incredibly elaborate tagging conventions, plugins, and scripts to organize their output along a bewildering number of dimensions. If you wanted to read a 3000 word fic where Picard forces Gandalf into sexual bondage, and it seems unconsensual but secretly both want it, and it's R-explicit but not NC-17 explicit, all you had to do was search along the appropriate combination of tags (and if you couldn't find it, someone would probably write it for you). By 2008 a whole suite of theoretical ideas about folksonomy, crowdsourcing, faceted infomation retrieval, collaborative editing and emergent ontology had been implemented by a bunch of friendly people so that they could read about Kirk drilling Spock.

Fan culture is extremely collaborative, and its participants had rapidly taught one another how best to combine LiveJournal, Delicious and other sites into an network for sharing and discovery that, due to the social stigma of the hobby, remained under the radar even though it would have meant instant success for any entrepreneur sincerely willing to work with them. Fans shared their setups and workflows with each other in much the way that startup subculture obsesses over tool chains and "stacks". The whole thing reminds me a lot of what HyperCard was like in the nineties, right before its demise, when a large number of otherwise non-technical users had basically taught themselves to do elaborate programming with the tool, and were doing amazing things with it. Until, of course, Apple pulled the plug.

Avos did a similar thing last week when they relaunched Delicious while breaking every feature that made their core users so devoted to the site (networks, bundles, subscriptions and feeds). They seemed to have no idea who their most active users were, or how strongly those users cared about the product. In my mind this reinforced the idea that they had bought Delicious simply as a convenient installed base of 'like' buttons scattered across the internet, with the intent of building a completely new social site unrelated to saving links.

For any bookmarking site, the fan subculture is valuable because it makes such heavy and creative use of tagging, and because they are great collaborators. I can't think of a better way to stress-test a site then to get people filling it with Inception fanfic. You will get thoughtful, carefully-formatted bug reports; and if you actually fix something someone might knit you a sweater. And please witness the 50 page spec, complete with code samples, table of contents, summary, tutorial, and flawless formatting, the community produced in about two days after I asked them in a single tweet what features they would want to see in Pinboard*. These people do not waste time.

They're also smart enough to realize that in the long run, they need to take control of their own bookmarking in order to avoid these kinds of surprises. One way they've done this is through what Britta calls the "friendliest goddamn open source project ever", an Archive of our Own. But this project is not quite ready to carry the full burden of everyone's bookmarks.

Pinboard is not a social site, and it has always been about archiving, not sharing. I don't intend to make the same mistake Avos did and suddenly try to retool the site for a brand new group while neglecting the quiet link hoarders who form the Pinboard old guard. As a grouchy hermit, I like to think that other grouchy hermits should have a place to store stuff that will never feel like publishing or expose them to unwanted contact with other people.

At the same time, I think the fans are a very nice bunch who have been somewhat hard done by, and that their presence will be a long-term boon to the site. Like bees in a garden, the sudden arrival of a big swarm can be alarming, but all this swarm wants is a place to set up a hive and get to work. And I'll end this metaphor right now before it provokes any pollination slash.

In the long run it's healthier for the site to have a more varied user base, and this is a group of users with a terrific track record of constructive feedback on other sites (notably LiveJournal) and a committment to privacy and non-commercialism that I think a lot of our original users share.

In a word: don't be afraid. Nobody has pivoted, nothing is changing. Just be careful what you click on for a while.

I'll post in a couple of days about what I'm planning to work on this fall, including my response to that epic feature spec. In the meantime I'll be hanging around Twitter and on pinboard-dev, as ever.

* naturally, they also produced this.

—maciej on October 02, 2011

The Costs of Bookmarking

I spent some time this morning working up my Pinboard expenses, and since I know this stuff is of interest to entrepreneurial types who are considering running their own service, I thought I would share the numbers here.

Some context to help make this information more useful:

• Pinboard is a service with about 15,000 active users (defined as people who get up in the morning and actually make the little lights on our servers blink). I am sure there is a more optimal way to run a service of this size. However, since the numbers you see on the spreadsheet represent real American dollars that I pay out of my own pocket, there is at least some amount of effort put into being frugal.


• The 'hosting' category covers a number of servers, from opulent production machines to little tiny computers that host source code or the site status page. You'll notice that I host with a whole bunch of people. Some of these represent great deals I got on individual servers, others are efforts to evaluate or maintain relationships with various hosting companies. And there is 'Joe's Datacenter', which has such an awesome name I just had to do business with them (and have been very happy).


• In August I moved the site from the unsavory DigitalOne to LeaseWeb, so there is a spike in costs reflecting the overlap.


• Some of the hosting services bill monthly, trimonthly, or annually, so the amount of money I spend in a given month on hosting may not correspond to the amount of hosting I get.


• The spreadsheet does not include generic costs of self-employment, like paying my own health insurance or tax prep. That stuff is amply documented elsewhere; please remember to think about it when considering going into business for yourself.

As you can see, not counting my labor it takes about $2k a month to operate Pinboard. If you have any questions about this stuff, please feel free to ask me on Twitter (@Pinboard) or by email.

—maciej on September 11, 2011

A Short Rant About Hosting

This weekend I finished moving Pinboard to our new web home with LeaseWeb in the Netherlands. In the course of looking for a hosting company I made up a spreadsheet of high-memory servers and their relative pricing. I focused on places that offered dedicated servers with over 32GB of RAM, since that is the kind of server Pinboard runs on. I've shared the information here because assembling it was such an enormous hassle (see rant below). Please email me (support@pinboard.in) if you have updates or know of hosting companies that I missed.

While I was looking for hosting, I heard a couple of recurring questions from users and potential hosting companies:

Why not go with Linode/AWS/[other virtualized hosting]?

My own experience with virtualized hosting is that I/O performance can be very poor, and that it can be impossible to diagnose underlying I/O problems from within the virtual machine. For example, in early days of Pinboard we found ourselves hosted on a machine that needed to recover a RAID volume. This so crippled I/O that it forced the site onto a backup server for three days, and yet from the perspective of the virtual machine everything looked the same. The 'physical' device was still mostly idle, just a hundred times slower than it used to be. That and similar adventures cured me of any desire to host on a non-dedicated server.

It's very possible that I'm doing this wrong and that VMs are a terrific way to host an online service. Lots of people love Linode and AWS. Like everyone else I'm a slave to my own particular experience and limitations.

Can you email our sales rep with your needs and your budget?

No. It's 2011 and this stuff is a fungible commodity. If I give you a specific server configuration, you can give me a price. If you're offering dedicated servers for rent but force me talk to a sales agent to find out the price, then I know you've just padded my rent with the cost of that person's salary. And if you ask me again what my budget is I swear I will smack you.

Quick quiz: can your entire sales staff be replaced by a nicely formatted HTML table? If the answer is 'yes', then you are subtracting value and wasting my time.

You can always find a lower price, but only we offer the Triple Titanium Serenity Rabid Gold SLA with our industry-shattering quadruple uptime guarantee!

If your server enclosures are made of elemental silver and polished daily with static-free cloths by computer science PhDs who are related by blood to Edgar Dijkstra, then I definitely want to know about that. Put a paragraph about it right under your price chart.

Realistically speaking, datacenters are complex systems. I doubt there's a more professional ops staff than at AWS, and yet even they have had outages and downtime due to unexpected events. I want to see that you have some kind of track record and clients that I've heard of, I want to know that you take sane precautions (multiple paths to the internet, not building data centers where an earthquake is certain to hit, backup generators, a happy and professional ops staff, a Twitter account for status updates), and I want to know that I can contact support and get a prompt reply from someone techical. But beyond that, there's no way I can really evaluate how awesome you are. So sell that hosting like the commodity it is!

—maciej on August 25, 2011

Two Years of Pinboard

Two years ago I started taking money from people in return for no-frills bookmarking. The signup price was rigged to grow by a fraction of a penny with each new user; it started at $2.04 for our very first customer (who immediately demanded a refund) and quickly soared to near the three dollar mark.

I posted our vital signs last year on this day; here's an update from twelve months later:

• 27 million bookmarks
• 12,500 active users
• 16 million URLs
• 3 TB of archived content
• 99.8% uptime (29 hours offline)

These stats include the two big events of the past year. The first came in December, when Yahoo accidentally catapulted us into prominence with a single powerpoint slide. The second came just a few weeks ago, when the FBI raided our data center and took off with an enclosure that included our main database server, in the process ruining my plans to boast about uptime.

This was a lot of excitement for one year. Since the whole point of Pinboard is to be around for the long haul, I'm hoping for much less of it in the year to come. A personal archive works a little bit like a bank - you want to be able to use it and forget about it, rather than see it pop up in the evening news. So I'm very grateful to all our users who reacted to both good news and bad this year with encouragement and kind words, and who showed great patience and understanding during the occasional outbreaks of drama.

No birthday post would be complete without the laundry list of new features we've added this year:

• Twitter integration with oauth
• Keyboard shortcuts
• French, German, and Japanese versions of the site
• Full-text search (for archival users)
• Tweet search
• Filtering by multiple tags
• Browser plugins
• Better tag auto-complete
• Instapaper integration (thanks Marco!)

In the year to come I'm looking forward to adding some pleasant new features, making the site more resilient to datacenter trouble, and doing what I can to make sure I'll be writing these birthday blog posts for many years to come. Thank you all for making this year such a triumph!

—maciej on July 09, 2011

Seeing Red

One of the things I've always obsessed about with Pinboard is speed. Bookmarking is like search in that when it gets sludgy, people start to use it less and less. I've taken pains since we launched to make sure we are faster than any other bookmarking site, sometimes by a lot.

But here's what I was looking at for most of the past week:

The numbers in the top section show page load times in milliseconds binned by the last minute, 10 minutes, and 60 minutes. This repeats three times: the first column shows the absolute slowest page load in that time period, the second shows load time at the 10th percentile (so 90% of load times were better), and the last one is the median value.

Below that is an internal server error where a whole piece of my admin page was unable to load. Below that are queue stats showing how many tag clouds have not been generated, and how many outside feeds are overdue for download.

Here's what things look like this weekend, now that normal service has been restored:

I'm going to spend the next couple of weeks making sure I never see that much red again.

—maciej on June 26, 2011

FAQ about the recent FBI raid

What happened?

Pinboard runs on servers in a Virginia datacenter. On Tuesday the FBI raided the data center and confiscated some computer equipment from our service provider, DigitalOne. This equipment included our main database server. The server was returned about a day later and is back online.

Pinboard was offline for several hours during the raid, and ran with reduced capabilities for about four days after that.

More details here and here.

Are my bookmarks safe?

Yes. Our servers are set up so that if one fails or goes offline, the other has a full copy of the data. (For the technically minded, we use a master-master replication setup with writes going to a single master). When the FBI took the main database server away, there was an up-to-date copy available on the backup server. We also continued making daily database backups to Amazon S3 during the outage.

Why did the FBI take a Pinboard server?

I don't know. As best I can tell, the FBI was after someone else whose server was in physical proximity to ours. The kind of computer we lease is a 'blade server', which means there are about a dozen of them arranged in each computer enclosure, like books on a shelf. If the FBI took an entire enclosure (or more), it would explain why so many servers went offline. I also understand it is routine in raids like this to remove as much equipment as necessary to preserve the chain of evidence and prevent suspects from covering their tracks.

It is not possible to tell at this point whether the scope of the raid was excessive. I'm trying to find out the details of what happened and will share whatever I learn.

Does the FBI have my bookmarks?

In order for the FBI to collect this data, they would need a warrant for it. At this point I have no reason to believe our site was included in the warrant. A FOIA request for the warrant has been filed and we should be able to see it within about two weeks.

What was on the confiscated server?

The server had a full copy of our database, the Pinboard source code, and contained archived web pages from about half our archival users.

Is my password safe?

This depends on how much you trust the FBI. Pinboard passwords are stored as salted SHA1 hashes, so you should assume that the FBI could easily crack your password if they wanted to. I'm in the process of moving to a safer form of password encryption (bcrypt) but at the time of the raid the user table still contained both forms of encrypted password.

An instapaper server containing Pinboard passwords was also seized during the raid. Those passwords were (for all practical purposes) stored unencrypted. If you connected your Instapaper and Pinboard accounts at any point you should change your Pinboard password wherever you use it.

Will bookmarks that I added on the backup server stay in my account?

Yes.

Is it still safe to use Pinboard?

It is as safe as it was before the raid. Our top priority is to avoid losing user data. But please, always make backups.

How can I be sure this won't happen again?

I'll be making several changes to how the site is hosted to make sure a data center failure can't affect us this badly again, and I'll post about them here shortly.

How can I get my data off of Pinboard and close my account?

Use the export page to grab your bookmarks, then send me an email.

—maciej on June 25, 2011