[Resolved] 2025-06-25 - Service Disruption "Required dependencies are missing from Connection!"

Root Cause Analysis (RCA)Date of Report: 2025-06-25

Summary

On June 24, 2025, some Riva Cloud customers experienced synchronization failures related to missing dependency information. The issue was traced to a recent update that unintentionally altered data required by the sync process to locate dependencies. Affected users encountered policy errors and interruptions to expected sync behavior. The issue was identified, triaged, and fully resolved by 09:33 EST on June 25, 2025.

Incident Timeline

• June 24, 19:20 EST – Initial customer reports of sync issues were received.
• June 25, 02:30 EST – Support team triaged the issue and escalated to on-call engineering.
• June 25, 04:21 EST – Root cause was identified and remediation planning began.
• June 25, 06:03 EST – A fix was developed and deployed to restore impacted connections.
• June 25, 09:33 EST – Sync services confirmed restored for all affected customers.

Root Causes

A recent update to the Riva Sync included changes intended to improve system logging. However, this update unintentionally altered part of the connection data structure used by the sync service to locate dependency information. This caused the sync process to fail  when attempting to load required dependencies, resulting in service interruptions for some customer policies.

Actions Taken

• Developed and deployed a targeted script to restore affected connection data.
• Performed integrity checks across impacted environments to confirm full recovery.
• Monitored sync behavior across all tenants to verify service stability.
• Initiated internal review to identify gaps in testing and validation processes.

Next Steps

• Improve alerting to better detect and escalate widespread sync failures.
• Implement safeguards to prevent unintended changes to critical connection data.
• Conduct a post-incident engineering review to incorporate lessons learned.

Conclusion

The incident was caused by an unintended side effect of a routine update. We recognize the inconvenience and concern this may have caused and have taken swift corrective action. Preventative measures are being implemented to ensure better detection, faster resolution, and improved reliability going forward. We appreciate your understanding and continued trust in Riva Cloud.