July 22, 2024 - Microsoft Graph Users are receiving connections errors preventing sync

Root Cause Analysis for the Microsoft Graph outage reported on July 22, 2024

Summary

An expired client secret within our system caused an interruption in the connection to Microsoft Graph, resulting in a service outage for all customers using Microsoft Graph as their email connection. This led to synchronization failures for affected users.

Timeline

• July 21, 2024, 8:56 AM EDT: The issue began with users experiencing errors related to expired certificates.
• July 22, 2024, 9:53 AM EDT: Riva notified customers of the critical issue affecting all Microsoft Graph clients.
• July 22, 2024, 2:25 PM EDT: Riva identified and resolved the issue for US pods, restoring synchronization for affected clients.
• July 22, 2024, 5:40 PM EDT: The issue was resolved for all other regions, including, Canada, Europe and Australia. 

Root Cause

An expired client secret prevented Riva from successfully authenticating with the Microsoft Graph service, disrupting data synchronization.

Analysis

The outage underscores the critical importance of credential management within critical infrastructure. Expired credentials disrupt communication and functionality, impacting user experience. The swift identification and resolution by the Riva team minimized the downtime for clients in our US and CA data centers, followed by a broader fix for all regions.

Improvements Implemented

To prevent a recurrence, Riva has transitioned from using client secrets to certificate-based connections for Microsoft Graph authentication. This change allows for more efficient monitoring and management of credentials. Additionally, a rigorous certificate rotation process has been implemented to ensure the ongoing security and reliability of the service.