Errors when creating bookings
Incident Report for FlexBooker
Postmortem

Writing to the Booking DB SQL Server RDS instance was failing with a generic timeout error.

After investigation, we found that INSERTS were being locked, but by sleeping SPIDs. Killing the process caused the lock to roll to a different process, in an endless rolling lock cycle.

We discovered that there was a fault in one partition of the instance, which we failed over to a new instance and rebooted, which cleared up the locks and allowed those bookings that had been made during the issue to complete their transactions and everything now looks ack to normal.

Posted Aug 04, 2022 - 15:03 UTC

Resolved
This incident has been resolved.
Posted Aug 04, 2022 - 14:58 UTC
Monitoring
The issue has been identified as a locked partition in the RDS instance used for bookings, which has now been unlocked and bookings are able to be made now. We will continue to monitor to ensure this resolution is final.
Posted Aug 04, 2022 - 14:48 UTC
Update
We are working with AWS to rectify an issue with an RDS partition and will update as the recovery process continues.
Posted Aug 04, 2022 - 14:35 UTC
Investigating
We are currently investigating an issue that new bookings are creating errors when being saved.
Posted Aug 04, 2022 - 14:04 UTC
This incident affected: FlexBooker Core (FlexBooker Staff Website, Booking Forms, API Services, Mobile Apps).