At 1647 BST on Wed 24 July we completed a routine release of the latest version of Virtuoso to our production environments. Part of this release removed an optional parameter on our web API that wraps responses with informational data (“Envelopes”). As part of the work to implement this, we consulted our monitoring and found that the parameter was omitted or set explicitly to false for all requests we had received over a substantial period. Removal of this parameter changed the response structure for requests that did not explicitly set the parameter as it is on by default, rather than off. This did not affect the Virtuoso interface as it disables envelopes.

Prior to all releases, the Virtuoso platform runs through a set of unit, integration, and end-to-end UI tests. Several tests related to the Bridge component failed, but it was incorrectly determined that this was an unrelated issue with the infrastructure that runs our test clients, and on that basis we proceeded to release.

At 1111 BST on Thu 24 July we were alerted to the issue and had a proposed fix deployed to our test environment at 1231 BST. This successfully fixed the issue, confirmed by running our whole test suite as well as further manual checks and this fix reached the production environment at 1328 BST.

We are sorry for the inconvenience caused by the incident. We have concluded our review of the incident and identified improvements to our processes that will safeguard against repeats of this incident. Changes to our API will require additional scrutiny, and our release process will require that any failing test cases require further review and manual confirmation before a release may proceed.