Microsoft details the causes of recent Multi Factor Authentication meltdown
Microsoft’s Azure team has gone public with the root cause it discovered when investigating the November 19 worldwide multi-factor-authentication outage that plagued a number of its customers. Actually, Microsoft unearthed three independent root causes, along with monitoring gaps that resulted in Azure, Office 365, Dynamics and other Microsoft users not being able to authenticate for much of that day.
For 14 hours on November 19, Microsoft’s Azure Active Directory Multi-Factor Authentication (MFA) services were down for many. Because Office 365 and Dynamics users authenticate via this service, they also were affected.
The first root cause showed up as a latency issue in the MFA front-end’s communication to its cache services. The second was a race condition in processing responses from the MFA back-end server. These two causes were introduced in a code update roll-out which began in some data-centers on Tuesday November 13 and completed in all data-centers by Friday November 16, Microsoft officials said.
A third identified root cause, which was triggered by the second, resulted in the MFA back-end being unable to process any further requests from the front-end, even though it seemed to be working fine based on Microsoft’s monitoring.
European, Middle Eastern and African (EMEA) and Asian Pacific (APAC) customers were hit first by these cascading issues. As the day went on, Western European and then American data-centers were hit. Even after engineers applied a hot-fix which allowed front-end servers to bypass the cache, the issues persisted. On top of all this, telemetry and monitoring wasn’t working as expected, officials acknowledged.
Microsoft identified a number of intended next steps to improve the MFA service, including a review of its update-deployment procedures (target completion date: December 2018); a review of monitoring services (target completion date: December 2018); a review of the containment process which will help avoid propagating an issue to other data-centers (target completion date: January 2019); and an update to the communications process for the Service Health Dashboard and monitoring tools (target completion date: December 2018).
Microsoft officials apologized to affected customers, but made no mention of any planned financial compensation. Microsoft’s November 19 Azure status history post has more details about the trail of events leading to the MFA meltdown.
*After this was published, Microsoft’s Azure Active Directory Multi-Factor Authentication (MFA) service went down for the second week in a row, on Nov. 27th.
Pulling the plug doesn't have to be your only security solution.
Don’t become part of a rising statistic — ensure your company is armed against a security hack.
Maureen Data Systems is a IT services and solutions company committed to developing custom, streamlined solutions for customers to achieve their business objectives.
MDS structures its highly skilled engineers to align with how our customers consume technology—with one team responsible for infrastructure, another heading up productivity and applications, and a third committed to identity and security solutions.
December 12, 2018 - NYC
Microsoft365 Interactive Workshop
January 16, 2019 - NYC
Backup & Get Your Systems Running with MDS & Veeam