At my employer, we use a service called Coveralls, which is a code coverage reporting tool that hooks into your CI pipeline. It takes your lcov reports and gives you some insights, and also makes it easy to cause the CI pipeline to fail if newly added code doesn't have reasonable coverage.
I have no strong feelings about the tool, but on February 24th, our CI jobs started failing. Coveralls had gone down, and our CI step that talked to them was the culprit. That happened to us, and a lot of other people. It was a widespread outage impacting anybody who used their service.
Coveralls actively published updates as the outage dragged on, and boy, did it drag on. Between them posting nearly identical updates minutes or even seconds apart and the general desperation that snuck into their outage notes after the first few days, I couldn't help but find their notes hilarious.
They've published their post-mortem, and the short version is that despite them paying their bills, Amazon decided they hadn't, and shut down their services. But it took them roughly 68 hours to figure that out.
If you scroll down past the post-mortem, you can see their outage notes. This is an unauthorized, dramatic reading of those notes. We've taken some liberties in interpreting it.
I hope this is taken in a good spirit, because they were just doing their best when this happened. I don't think most companies would be able to maintain operations when their cloud provider decides to lock their accounts. But I don't care who you are: begging your users to email you if they can help you migrate off AWS is funny.
[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!