Microsoft details the causes of recent Multi Factor Authentication meltdown


Microsoft’s Azure team has gone public with the root cause it discovered when investigating the November 19 worldwide multi-factor-authentication outage that plagued a number of its customers. Actually, Microsoft unearthed three independent root causes, along with monitoring gaps that resulted in Azure, Office 365, Dynamics and other Microsoft users not being able to authenticate for much of that day.

For 14 hours on November 19, Microsoft’s Azure Active Directory Multi-Factor Authentication (MFA) services were down for many. Because Office 365 and Dynamics users authenticate via this service, they also were affected.

The first root cause showed up as a latency issue in the MFA front-end’s communication to its cache services. The second was a race condition in processing responses from the MFA back-end server. These two causes were introduced in a code update roll-out which began in some data-centers on Tuesday November 13 and completed in all data-centers by Friday November 16, Microsoft officials said.

A third identified root cause, which was triggered by the second, resulted in the MFA back-end being unable to process any further requests from the front-end, even though it seemed to be working fine based on Microsoft’s monitoring.

European, Middle Eastern and African (EMEA) and Asian Pacific (APAC) customers were hit first by these cascading issues. As the day went on, Western European and then American data-centers were hit. Even after engineers applied a hot-fix which allowed front-end servers to bypass the cache, the issues persisted. On top of all this, telemetry and monitoring wasn’t working as expected, officials acknowledged.

Microsoft identified a number of intended next steps to improve the MFA service, including a review of its update-deployment procedures (target completion date: December 2018); a review of monitoring services (target completion date: December 2018); a review of the containment process which will help avoid propagating an issue to other data-centers (target completion date: January 2019); and an update to the communications process for the Service Health Dashboard and monitoring tools (target completion date: December 2018).

Microsoft officials apologized to affected customers, but made no mention of any planned financial compensation. Microsoft’s November 19 Azure status history post has more details about the trail of events leading to the MFA meltdown.

*After this was published, Microsoft’s Azure Active Directory Multi-Factor Authentication (MFA) service went down for the second week in a row, on Nov. 27th.

**This article was written by Mary Jo Foley for All About Microsoft on Nov. 26, 2018. The original article can be found here.

Pulling the plug doesn't have to be your only security solution.

Don’t become part of a rising statistic — ensure your company is armed against a security hack.