Major DDoS attack affecting all servies
Incident Report for FlexBooker
Postmortem

FlexBooker June 20th DDoS Post Mortem

What happened?

Starting at 12:30 on June 20h 2024 a distributed denial of service (DDoS) attack began hitting several FlexBooker services at increasingly high rates, leading to diminished performance (longer API calls and page loads). FlexBooker reached out to Amazon AWS (the hosting company) for assistance. This led to Amazon's initial reaction to begin terminating Internet-facing network adaptors, leading to a total outage. The attack only hit public-facing endpoints, and created service and app overloads, but at no point was any data (merchant or client) ever exposed.

Resolution

After initially working with AWS to block the attack to turn the network adaptors back on, we felt that the support response times were too slow. This led us to explore other options while also pursuing AWS support. A complete migration of a new environment to Microsoft Azure was initiated, along with all app services, queues, data, and cache layers. This was completed on the evening of June 25th. DNS aliases were switched to Azure, which over the proceeding hour or two of propogation led to the restoration of all services.

Corrective Actions

The Azure environment was set up in close coordination with Microsoft Azure security and support to ensure resistance from future DDoS attacks, including tuned Azure DDoS Protection. The AWS environment has been restored also, which will be kept as a completely independent emergency environment.

Posted Jul 17, 2024 - 19:33 UTC

Resolved
This incident has been resolved.
Posted Jul 17, 2024 - 19:30 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 30, 2024 - 22:19 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jun 21, 2024 - 05:07 UTC
Identified
We are currently experiencing a massive DDoS attack. It started at a small level which we were able to handle but has grown into a huge distributed targeted attack. We are working with our service provider (AWS) to try to resolve this, and also looking into other options to bring things back as they work on this.
Posted Jun 21, 2024 - 03:39 UTC
This incident affected: FlexBooker Core (FlexBooker Staff Website, Booking Forms, API Services, Mobile Apps), Customer Support Chat (Intercom Mobile Messenger, Intercom Chats and Posts, Intercom Web Messenger, Intercom Intercom Web Application, Intercom Email), Email Messaging (mailgun API, mailgun Outbound Delivery, mailgun SMTP), Text Messaging (Twilio SMS Long Code, North America, Twilio MMS Long Code, North America, Twilio SMS, Europe, Twilio SMS, Twilio MMS, Twilio REST API), and Payments (Braintree United States Processing, Stripe API).