Screenshots are failing to be captured by our bots during executions
Incident Report for Virtuoso
Postmortem

At around 16:41PM we started receiving error alerts indicating that screenshots and other captured data (downloaded files, network logs) could not be saved to our object storage backend. At around 17:11 we identified the cause and rolled out a fix by 17:18. While screenshots were not stored for some steps for jobs running within this period, other execution results were still collected, and executions continued uninterrupted.

We recently changed the authentication mechanism that our execution backend uses to upload screenshots and other information during execution. The team released the latest platform to production today which included this change. However, our release automation did not make the necessary infrastructure changes at the same time which led to the failed authentication events.

Posted Feb 06, 2023 - 17:37 UTC

Resolved
This incident has been resolved.
Posted Feb 06, 2023 - 17:25 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 06, 2023 - 17:19 UTC
Identified
We have identified an issue where for a small subset of browsers, screenshots are failing to be persisted.
Posted Feb 06, 2023 - 16:41 UTC
This incident affected: Jobs and bot cluster.