Speculating On Outlookageddon
During the recent Microsoft outage, where access to Outlook and associated services were seriously degraded for almost 24 hours, there was a lot of speculation about what might have caused the issue. We’ll have a lot of answers when the postmortem is published, but it’s always fun to explore theories from a point of uninformed ignorance (a position from which I often do my best work), and every now and then something interesting comes out of these exercises. I also thought it would be good to tie some of these ideas back to things I cover in Building Resilient Distributed Systems (new book, out now) because well, I wrote the book as I want people to read it.
One theory (OK, it's one of mine!) was that the prolonged outage could be caused by some sort of circular dependency issue, making it difficult to bring services back online. At a simplistic level, imagine that A->B->C->A. If A goes down, restarting it may be difficult or impossible, as A depends on B, which in turn depends on C, but C won’t work because A is unavailable. I should probably come up with a fancy name for this issue like "The Ouroboros Effect" or something, but that sounds too much like a Robert Ludlum book, so I for the moment I'm describing it as a circular bootstrapping issue.
In a massive-scale system, like the ones we can imagine run behind the scenes at Microsoft, it’s easy to imagine that some circular runtime dependencies like this could exist, and may not be entirely visible to the operators. Turning off large parts of the system to test for circular bootstrapping problems is a non-trivial affair. Remember that in many cases services are often never entirely taken down, with rolling updates used to roll out changes, meaning that some services may not have been entirely offline for months or perhaps even years. In such a situation it’s easy to imagine that you might have a potential lurking circular bootstrapping issue just waiting to happen in your own infrastructure.
So how do you avoid a circular bootstrapping issue like this? Well, the easiest answer is to restructure the system to remove the circular dependency in the first place. In the case of A->B->C->A, splitting responsibilities into separate services can help (e.g. splitting A -> A1, A2, resulting in A1->B->C->A2). This though can involve a lot of work, and you have to know the problem is there. So what else could you do to defend against this issue?
One thing that could help is to ensure that a service can be launched without its dependent services being available. If A->B, when launching A, if B is unavailable, then A should be able to be launched in a potentially degraded state. If some functionality that A provides does not need to use B, then that functionality should be made available to the clients of A if possible. This puts A into a partial failure mode of course, so you would have to give considerations as to how client circuit breakers would behave - I explore a bunch of ways to handle partial failure and other issues with circuit breakers in chapter Chapter 6 of Building Resilient Distributed Systems (BRDS).
When a service goes into a partially degraded state due to one of its dependencies having an issue, you'll also need to ensure the service has the ability to recover when their dependencies are healthy again. You could centrally manage this with a control plane or similar, or use some out-of-band comms between client and server, but sending a small number of calls via a client-side circuit breaker or token bucket could be enough to establish that it's dependency appears to be healthy again.
Another problem which can lead to prolonged downtime is an internal thundering herd that cannot easily be stopped. In Chapter BRDS I give the example of a Square outage back in 2017 which unleashed a thundering heard. Square's multipass authentication system was restarted in an attempt to help mitigate an incident that was being managed - unfortunately this restart made things worse. Multipass stored its data in redis. When restarted, all the multipass nodes started hammering redis, soon exhausting connections. Rather than backing off to let redis recover, the hard coded logic in multipass would ensure that 500 retries would be attempted without any delay between attempts, before giving up. As a result of this incident Square fixed the retry logic (see Chapter 4 in BRDS for more). In the case of multipass, it was on the critical path to user access - so it being unavailable was a major problem.
Imagine that you have hundreds of clients of a key service, and when that key service goes offline and comes back up, all those clients start hammering the newly revived service with calls. To avoid this you need defensive rate limiting techniques on the server side to ensure that excess work is either dropped entirely (load shedding) or apply back pressure to ensure that clients back off. In a large scale system you cannot assume that clients will be well behaved - client-side techniques like circuit breakers and token buckets are typically optimizing for the local client, rather than thinking about overall system health. Having a service endpoint shed excess load as a defensive measure ensures that some work gets processed, rather than trying to process everything and then fail.
In some cases, refusing all work until dependent services are available may be the right call. Imagine a scenario where your service depends on a caching node for fast reads. The cache is restarted. If you still allow for reads, the now-empty cache means that reads could now start hammering the origin. Waiting until the cache is re-populated again may be the right call - refusing some (or all) calls now to stop an initial problem becoming a progressive collapse (see BRDS chapter 11) is going to be the smart play.
Whether or not the above issues are in the mix with the Microsoft outage is something we’ll find out when the postmortem is made public. Once it’s out, I’ll definitely make sure to take a look and share my thoughts, mapping some of the concepts I detail in BRDS to this real-world example.
But for now, cut the folks at Microsoft some slack. They’ve had a pretty torrid experience, and they’ll need time and clear heads to carry out a cogent post-incident review. When things like this go wrong, I like to spend my time working out what I can learn from this, and how the systems I help with could improve based on what we learned, rather than throwing shade on the poor folks dealing with a major incident. And if you want to learn more about how to make your own system more resilient, then Building Resilient Distributed Systems is out now!