Root Cause Analysis (RCA)
Date of Report: November 11, 2024
Summary
This report covers recent performance issues in the USE1 region of Riva Insight, which impacted reliability and user experience. The main causes included increased user load, CRM cache inefficiencies, and technical delays in data processing. Immediate fixes were applied, and further improvements are underway to ensure a stable and reliable service.
Incident Timeline
- Early October: Monitoring identified performance issues.
- October 7: Performance reports received, prompting immediate action.
- October 8-22: Infrastructure upgrades completed, including server and software updates.
- October 30 - November 6: Additional performance optimizations implemented.
Root Causes
- The performance issues were caused by a surge in user activity, combined with some system inefficiencies, which together impacted the platform's ability to handle the increased demand smoothly.
Actions Taken
- We implemented improvements to support higher demand, increase speed, and streamline data processing for better performance.
Next Steps
- Our next steps include strengthening monitoring, preparing for growth, optimizing data queries, and testing to prevent future delays.
Conclusion
We understand the impact of this issue on your operations, and we are fully committed to providing you with a high standard of service. Immediate actions have been taken, and further improvements are in progress to ensure reliability. Thank you for your understanding as we work to strengthen our platform’s stability and performance.
(Previous updates)
We have received multiple reports of performance-related issues and general slowness with Riva Insight on the USE1 server.
The Team is continuing to work and identify areas where improvements can be made to performance. Another round of updates is in the works, additional information to follow.
November 05, 2024
We have identified several performance issues and implemented improvements to address them. The new version is now in testing, showing marked improvements compared to last week. We anticipate deploying this version this evening. While additional work remains, this is a positive step forward.
November 04, 2024
Over the weekend, we expanded some infrastructure to better scale.
November 01, 2024
New monitoring tools were added to better identify bottlenecks.
October 07, 2024
We have received multiple reports of performance-related issues and general slowness with Riva Insight for users on the USE1 server. Our team is aware of the issue and actively investigating.
Riva insight users on other data centers or dedicated instances are not impacted.
Additional updates will be added as they become available.