Cloudy Snake Oil

I have nothing against Amazon S3, but I was astonished today to find this claim on their FAQ page:

Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities.

This is an impressive number, but it's utterly dishonest to make such claims. It implies that there is a less than one-in-one-hundred-billion chance that Amazon will abruptly go out of business, or that a rogue employee will cause massive data loss, or an unexpected bug will result in massive data loss, or a defect in storage media will cause millions of devices to fail silently, or a large solar flare will destroy equipment across three data centers, or that a comet impact will destory three data centers, or that a nuclear exchange will destroy three data centers.

I think these events are all incredibly unlikely, but none of them is one-in-a-hundred-billion unlikely. Yet here is Amazon not only making that argument, but implying that you can safely use S3, a service that launched in 2006, for another ten million years.

Rare events are rare! That's why promises past five or six nines of reliability are functionally meaningless. At that point the "unknown unknowns" must overwhelm any certainty you have about what you think your system is doing.

The risks you failed to model will become obvious in retrospect, and make for an entertaining post-mortem, but that won't get anybody's data back.

Promises like Amazon's should serve as a kind of anti-marketing, suggesting that the company has not thought seriously about the limits of risk assessment and planning.

I suggest the following rule of thumb: if you can't count the number of nines in the reliability claim at a glance, it's specious.

Of course this rant is available in book form, phrased better than I have here. But it's worth repeating at every opportunity.

—maciej on April 19, 2014