Errors when creating bookings

Incident Report for FlexBooker

Postmortem

Writing to the Booking DB SQL Server RDS instance was failing with a generic timeout error.

After investigation, we found that INSERTS were being locked, but by sleeping SPIDs. Killing the process caused the lock to roll to a different process, in an endless rolling lock cycle.

We discovered that there was a fault in one partition of the instance, which we failed over to a new instance and rebooted, which cleared up the locks and allowed those bookings that had been made during the issue to complete their transactions and everything now looks ack to normal.

Posted Aug 04, 2022 - 15:03 UTC

Resolved

This incident has been resolved.
Posted Aug 04, 2022 - 14:58 UTC

Monitoring

The issue has been identified as a locked partition in the RDS instance used for bookings, which has now been unlocked and bookings are able to be made now. We will continue to monitor to ensure this resolution is final.
Posted Aug 04, 2022 - 14:48 UTC

Update

We are working with AWS to rectify an issue with an RDS partition and will update as the recovery process continues.
Posted Aug 04, 2022 - 14:35 UTC

Investigating

We are currently investigating an issue that new bookings are creating errors when being saved.
Posted Aug 04, 2022 - 14:04 UTC
This incident affected: FlexBooker Core (FlexBooker Staff Website, Booking Forms, API Services, Mobile Apps).