Resolved - This incident was resolved at 10:15 AM UTC after approximately 75 minutes of downtime.
The root cause was a dependency update. We had rolled out incrementally for >6 hours without incident, but hit an edge case in error handling that resulted in an unhandled exception. We mitigated the issue with a forward fix and will review our policy for rolling out dependabot upgrades. (10:28 UTC — Dec 31)
Monitoring - We’ve fixed the issue and monitoring the system. The issue was related to an edge condition around latest dependency security update that didn’t surface during the daytime testing and monitoring. (10:19 UTC — Dec 31)
Investigating - We are investigating an outage (09:54 UTC — Dec 31)