World of Pinboard

Slowly making my way home to Poland for a visit gave me a chance to meet a number of Pinboard users along the way. I am grateful to everyone who took the time to join me for the impromptu meetings documented below. I've found that meeting actual human beings who use my site is a valuable weapon in the fight against burnout. But it's been even nicer to discover that some really nice people use Pinboard. This encourages me to try to be a nicer guy myself.

Refusing to stay in focus in Warsaw:

Conquering Lyon with @anatsuno:

Friendly Pinboard people in Paris:

—maciej on February 14, 2013

Warsaw Meetup Today

A reminder that Warsaw Pinboardites are invited to meet me today at 4 PM at the Amatorska café on Nowy Świat 21. Previous announcement here.

—maciej on February 09, 2013

Paris Meetup

I'm organizing a meetup Saturday, January 26 for Pinboard users in Paris. Send me email if you'd like to attend.

—maciej on January 25, 2013

Pinboard Co-Prosperity Winners

Today I am delighted to announce the six projects I have selected for the inaugural round of the Pinboard Co-Prosperity Cloud.

I received 306 applications in total, many of them extremely strong, and narrowing them down to six was not easy. It meant saying no to some very good ideas, and I hope that many people who received a rejection will show me up by creating a successful small business anyway.

The six finalists represent a mix of projects that I think are tractable, useful, clever, and capable of bringing their creators financial independence. They are, in no specific order:

Alex "Skud" Bayley, Australia, Growstuff

Growstuff is a website where food gardeners will be able to track and share their food-growing efforts. A year ago I would have thought Skud's proposal a little idealistic, with its emphasis on process, collaboration, radical openness, and inclusiveness. But my brush with fandom a year ago (when a distributed group of volunteers responded to my call for feature requests by drafting a beautifully organized 50+ page Google doc in about 48 hours) has made a true believer out of me. Moreover, I think food gardening is a natural fit for the kind of community-first approach Skud wants to pursue. I jumped at the chance to pick a project coming out of this friendly, highly collaborative world, and I can't wait to see what it grows into.

Luiz Irber and Guilherme Castelao, Brazil. Low-bandwidth weather forecasts for sailors

If you've ever had to deal with marine weather forecasts, you will immediately see the appeal of this idea. Marine forecasts are poorly organized, often presented in graphical form only, and hard to get where you need them most (on the boat itself, where Internet access costs a fortune). Guilherme and Luiz want to package publicly-available data on currents, wind, and weather into a succinct forecast that sailors can have sent to them by email.

Right now, customized forecasts for boaters exist only at the luxury end of the spectrum, for people who can afford to go out on the water encrusted with expensive electronics. I particularly like that Luiz is a computer engineer in earth sciences, and Guilherme has a PhD in oceanography and is an avid sailor. These guys clearly know what they're up to, and their only remaining barrier to success has been a lack of $37.

Nik White, United States, Simple Cyber Security

Nik is a security guy who wants to offer pre-hardened machine images for cloud services like EC2, Rackspace and VMware. This is such an obviously useful idea that I can't believe it has not been done yet. Like in many areas of computer security, there is no middle ground right now between 'do it yourself for free' and 'pay tens of thousands of dollars to hire an expert'. Securing your own system is time-consuming and very easy to get wrong, and for some projects, you need to meet specific standards. Having canned, hardened AMIs from a trustworthy source fills this gap in the market, allows Nik to partially automate his expertise, and guarantees him a flow of consulting work as an adjunct to his main idea.

Lucas Howell, United States, Board game project

Lucas wants to build a site for fellow board game enthusiasts who want to keep track of their games, remember defeats and triumphs, and compare strategies. Right now this market is dominated by an old and somewhat sprawling site called BoardGameGeek, where the emphasis is more on discovering new games than keeping track of ones you've played in a structured format. BoardGameGeek is also plagued by performance issues and downtime. Lucas thinks there's a good opportunity here for a more focused site, and since these circumstances are almost exactly analogous to the ones that made me create Pinboard as a scrappy alternative to Delicious, I found his pitch persuasive. I also like the fact that Lucas is building something he really wants to use himself.

Bernard Huang, United States. Food By People

The idea behind food by people is a friendly and appealing one: let people sell their home baked goods online. Right now because of antiquated food safety laws this is only legal in 31 states. I believe this is an area that is ripe for regulatory reform, and I'm pleased to support a project that will push at those boundaries. If we can have quasi-legal pot sold online, then the complementary market for baked goods is obviously going to explode. Bernard already has two bakers using the site and making money, always a promising sign. I like the focus of his idea, and I like the idea of not having to pay a fortune to order a box of delicious cookies online.

Indy Griffiths, New Zealand, Parent Interviews

As Indy describes it, his project is "a website where schools sign up, set up an interview evening, and then parents can log in and book their interviews." It belongs to the group of unglamorous but essential education websites that are currently dominated by crapware. In the US, we have big and awful market leaders like Blackboard.com. In New Zealand, Indy has an established competitor who charges schools a fortune for a terrible user expereince. What particularly impressed me about Indy's idea is that he built a full mockup as part of his scholarship project, and now he's ready to try to make it a viable business. As a recent student, he knows firsthand both what the market needs, and just what makes schools hate the current offering as much as they do. He's clearly thought the idea through and is ready to go forth and sell.

I'll have more to say about each project in a future blog post. Thanks again to everyone who applied, and the many wonderful sponsors who have offered to donate services the winners.

And thanks especially to Jesse Vincent, Diane Person, Ann Hess, Conrad Heiney, Jeff Smith, and Britta Gustafson for helping me with the final selection. Any errors I made are my own, but there would have been more without these nice people.

Congratulations to the winners!

—maciej on January 13, 2013

The Pinboard Co-Prosperity Cloud

For those of you who missed the announcement on Twitter, last week I launched a new startup auto-incubator, the Pinboard Investment Co-Prosperity Cloud. Six selected applicants will receive a tiny amount of funding, along with a generous amount of publicity, to help them launch their sustainable Internet business.

The response so far has been very encouraging. I've received just over 200 submissions, and the overall quality of ideas has been remarkably high. I'm amazed at the response so far, and grateful to everyone who has pitched in with kind words, sponsorship, or taken the time to submit a project.

Applications are open until January 1, and I hope to announce the winners about a week after that.

—maciej on December 26, 2012

Japan Meetup

I'll be in Japan until January 11. Please email me (maciej@pinboard.in) if you'd be interested in attending a Pinboard meetup in Osaka or in Tokyo. You don't have to speak English! We can just smile and enjoy terrific snacks.

I'll be in Osaka from January 4-8, and in Tokyo from January 8-11. I'm looking forward to meeting, and re-meeting, some of the wonderful Pinboard users here.

—maciej on December 20, 2012

Twitter and Pinboard

Yesterday, the popular glue website IFTTT removed the ability to perform actions based on Twitter events, due to their reading of recent changes to the Twitter API. A number of Pinboard users have asked me for clarification about what this means for Pinboard, which offers pretty extensive Twitter integration.

Right now, you can connect up to three Twitter accounts to Pinboard, and have the site automatically store your tweets and favorites. Pinboard also lets you export all your archived tweets in JSON format, and makes your tweets searchable. The service is subject to technical restrictions in the Twitter API (which won't return more than the last 3000 or so tweets), but if you set it up early enough, you can maintain an archive going back arbitrarily far.

Recently, Twitter announced a series of new restrictions on how third party services can display tweets to users. Among other things, the display restrictions would make it impossible to continue to make people's tweets searchable, or link them to associated bookmarks in their Pinboard account.

On August 17, I sent an email to Michael Sippey at Twitter, asking him to clarify Twitter's position on archiving:

Hi Michael,

I read your recent blog post with interest, and hope you can clarify the role of personal archives under the new API rules.

For context, Pinboard offers a Twitter archiving service. Users connect their accounts via OAuth and the site downloads and archives their tweets, along with associated bookmarks. These tweets are viewable and searchable on the site, but are only visible to the owner of the account.

Is it Twitter's policy that the tweet display rules be enforced when people are interacting with their own archived Twitter data, in a private client?

I am still waiting for a reply.

My position is straightforward. I believe users have the right to retrieve their own data from any third-party service for the purposes of personal archiving. I believe that they have the right to view and interact with this data without restriction, as long as that view is private to them. For this reason, I don't intend to restrict Pinboard users' ability to manage their archived tweets.

But I also think it's a little early to hoist the black flag. To me, the most plausible theory is that Twitter just didn't think about third-party backup when drafting the new document.

I am optimistic that Twitter will clarify their terms of service with regard to personal backups. In 2012, it remains impossible for a Twitter user to do a full export from their Twitter account. The only way to effectively store and search tweets is through third-party sites.

I don't believe Twitter wants to have the kind of adversarial relationship with their users that a literal reading of their new terms of service would imply, and I look forward to a constructive conversation about it with them.

—maciej on September 28, 2012

Every Day I'm Bundlin'

I've added tag bundles as an experimental feature. 'Experimental' means I'm going to futz with it without warning, and change the part you like best. Caveat bundlor!

A bundle is a named collection of tags. Say you have a bunch of bookmarks tagged orange, apple, and banana. You can create a bundle called fruit and have it display all bookmarks that have one of those three tags.

Bundles are most useful to people who have highly structured tags, or huge numbers of tags. But they're also handy for creating little thematic collections without having to edit lots of bookmarks at a time.

Here's a screenshot of the bundle editor:

The editor lets you filter tags by prefix, view only unbundled tags, and move tags in and out of a bundle either using the keyboard or by clicking with the mouse. There's no provision right now for seeing or filtering by a list of bundles within the editor, but part of my futzing is coming up with a version of the UI that can do that cleanly.

Here's what your list of bundles will look like in edit mode:

You can order your bundles in arbitrary sequence. You can also make bundles private by prepending a dot to the name, just like private tags.

I'll be adding bundle documentation to the howto page later this weekend. Meanwhile, you can turn on the tag bundles option on your settings page if you are feeling adventurous.

To get started bundling, enable the setting and head back to your user page. You'll see a 'no bundles' link above your tags; click it.

Please send me feedback, either via Twitter or email, and I'll do my best to fold it back in to the feature.

—maciej on August 10, 2012

Default Secure Connections

I've added a setting to the privacy settings page for users who want Pinboard to automatically use a secure connection. Turning it on will make Pinboard always redirect to an https URL. This setting is on by default for new users, and I encourage everyone to enable it.

People have asked about other logical steps (like setting the secure flag on cookies), and why it's even an option to use the site without SSL. I wholeheartedly agree with the sentiment, and these things are coming. However, given the large number of active users, there are bound to be edge cases, and I am proceeding incrementally in order not to hit them all at once.

I welcome all bug reports and ideas about how to better lock down the site. Extra bonus points if you send them by email, rather than all-caps announcement on Twitter!

—maciej on August 08, 2012

A Bad Privacy Bug

On Friday, I made a site update that exposed private bookmarks for about a hundred Pinboard users. Any private bookmarks created through the browser between about 15:45 and 17:30 Pacific time were visible on public areas of the site.

I emailed affected users on Friday evening to inform them of what had happened. If you didn't get an email from me, it means your bookmarks were not affected. But if you have any doubts, by all means get in touch and I will take a look.

After losing data or passwords, this is the about the worst kind of mistake I can make.

The heart of the problem was that it turned out to be possible to ask the Pinboard database to "give me only bookmarks where the privacy flag is set to zero'" and still get back results where the privacy flag was on. This is like accidentally baking something by putting it in your freezer. Unexpected.

In my email to users, I promised to give a more public and technical explanation of what I did wrong:

The proximate cause of the bug was a small change in the 'save bookmarks' page:

      - $b->private   = post("private") ? 1 : 0; 

      + $b->private   = post("private");

In human language, the original line says 'set the private flag on the new bookmark to zero or one, depending on whether there was a 'private' field in the submitted form', and the replacement line says 'set the private flag to either nothing, or the value the user posted'.

I made this change believing that there was code later in the save process that coerced the value to either 0 or 1. There wasn't. That was mistake #1.

So now a new bookmark coming into the system would have $b->private set to 'yes' instead of 1.[*]

That shouldn't have been enough to break things. Mistake #2 was in the way I set up the database, way back in 2009. Here's the database definition for the bookmarks table :

CREATE TABLE bookmarks (  
    id int(11) NOT NULL,
    url mediumtext,      
    title varchar(255),  
    description mediumtext,  
    user_id int(11) NOT NULL,  
    toread tinyint(1) DEFAULT '0',  
    private binary(1) DEFAULT '0',  
    url_id int(11),     
    slug char(20),       
    snapshot_id int(11),  
    code char(3),        
    source smallint(6),   
    added_at datetime,      
    created_at datetime,   
    updated_at datetime,  
    PRIMARY KEY (id),  
) ENGINE=InnoDB DEFAULT CHARSET=utf8;  
   

Notice that 'private' is a binary field of length 1. This was a braino on my part - I was thinking of the fact that I just needed to record a single bit. But the binary column type in MySQL is intended for binary data, not Boolean values. A more correct way to store this kind of flag is as tinyint(1). Notice that the toread flag has this correct type.

An important difference between these data types is how they behave when you put in character data:

 mysql> create table demo(int tinyint(1), bin binary(1));
 mysql> insert into demo(int, bin) values ('y', 'y');
 mysql> select * from demo;

 +------+------+
 | int  | bin  | 
 +------+------+
 |    0 | y    |
 +------+------+

Notice that saving a non-number value in the tinyint column converts it to zero.

This would have let me catch the bug in testing, since a bookmark saved as private would have been displayed as public (the two are visually distinct).

But this is the part that I really did not expect:

 mysql> select bin from demo where bin = 0;

 +------+
 | bin  |
 +------+
 | y    | 
 +------+

So even though the row has (from our perspective) a non-zero value, MySQL sees it as matching zero.

This means that a database query for public bookmarks (which includes a 'WHERE private=0' restriction) will still return bookmarks that have all kinds of values in the private field.

When PHP converts such a row into a Bookmark object, it assigns the privacy status based on the value it gets from the database:

   $b = new Bookmark();
   $res = query('huge sql query here');
   $row = $res->fetchRow();
   foreach ($props as $p) {$b->$p = $row[$p];}

At this point the $b->private instance variable is set to 'y'. Because of how PHP works, any Boolean test of $b->private will now evaluate to 'true', and so we have here the miracle of a truly private bookmark returned from a strictly public query.

So mistake #3 was not trapping unexpected data types at load time.

Now, there are certain pages on the site (like /recent/ and /popular/) where it should never be possible to see private bookmarks. Mistake #4 was not enforcing this restriction in the display template. This extra check exists for private tags, but it did not exist for private bookmarks.

While it would not have prevented this privacy leak entirely, it would have limited the number of pages where private bookmarks could show up.

Finally, mistake #5 had to do with how I test this stuff. Instead of automated test suites, I use checklists before deploying major changes, performing a series of actions (like creating and editing bookmarks) to make sure everything works as expected.

But I do this from my own Pinboard account, which has superpowers. Specifically, it lets me see private bookmarks on all accounts. And since I already see everything, I don't notice when everybody else starts to see everything, too.

tl;dr: because of poor input validation and a misdesigned schema, bookmarks could be saved in a way that made them look private to the ORM, but public to the database. Testing failed to catch the error because it was done from a non-standard account..

There are several changes I will make to prevent this class of problem from recurring:

1. Coerce all values to the expected types at the time they are saved to the database, rather than higher in the call stack.

2. Add assertions to the object loader so it complains to the error log if it sees unexpected values.

3. Add checks to the templating code to prevent public bookmarks showing up under any circumstances on certain public-only pages.

4. Run deployment tests from a non-privileged account.

Moreover, I'm considering adding special behavior for accounts that are private by default. If you really, truly never want to save public bookmarks, there should be a way to tell the site that in exchange for additional safeguards.

One thing I won't do fix is the database schema. While the field type is incorrect, at this point a whole pyramid of working code sits on top of it, and mucking with it might break other things in unexpected places. Prudence suggests not touching it without lavish preparation.

To everyone who was affected by this bug, or who could have been if the timing was different, I would like to offer my sincere apology. After spending three years seducing the privacy-obsessed, it's pretty low form to suddenly break those guarantees.

While I've tried to compensate affected users, it's no substitute for not allowing these kinds of mistakes in the first place.

I'm around on Twitter as usual if you have questions or things to say.

[*] This is not to say there was no data validation done at all. Incoming data is sanitized in a separate step.

—maciej on July 31, 2012