The incident was caused by an upgrade of an external dependency in the Virtuoso application. One of the updated versions introduced a regression that caused database connections to stall after about 24 hours of continuous activity. Users first experienced job delays and stuck jobs over the weekend, but the severity increased on Monday 26th January as platform usage rose. By replacing the affected compute instances at 9:06am GMT, the team temporarily restored functionality by 9:38am.

After the immediate fix, the team undertook monitoring and investigation, discovering that the issue consistently appeared after a 24-hour period. This was confirmed with the degraded performance on the 28th January, which allowed us to collect additional information. These insights allowed targeted data collection and more proactive interventions, preventing further episodes of degraded performance.

Eventually, root cause analysis led to a specific upgraded library being identified as the trigger for the behaviour. Updating to the latest version with additional fixes did not resolve the problem, but reverting to the previously working version resolved it.