Wasabi Managment Console MFA authentication issue seen

Incident Report for Wasabi Technologies

Postmortem

Between 10:32 UTC 2024-08-06 and 20:40 UTC 2024-08-07, we experienced three outages affecting both S3 and user services in all regions.

Starting at 10:32 UTC 2024-08-06, our queueing service reached a full capacity state which impacted our database cache causing it to become unresponsive. The Wasabi Operations team initiated a restart to the primary database in an attempt to clear out all stale connections to the database while simultaneously clearing the queuing service queue. When this action failed to bring the database into a fully operational state, the secondary database instance was promoted to primary. At 11:20 UTC the S3 service was fully operational again. Between 13:17 UTC and 13:23 UTC, the database was restarted once more by Operations in order to fully incorporate our queueing service library.

Between 02:55 UTC to 03:35 UTC on 2024-08-07, a second event occurred when our Operations team identified a configuration issue within the queueing service and the previously promoted secondary database instance. This configuration issue was causing timeouts to occur on user services such as our Web Console, WAC API, and WACM interface. Our Operations team then promoted the primary database back to production, alleviating these issues. There was no impact to S3 services during this event.

Between 20:30 UTC to 20:44 UTC on 2024-08-07, a third event occurred when an automation cluster was failing to be seen by our automation service, causing a small decrease in accepted traffic to our S3 vaults. Our Operations team then recreated and redeployed this cluster, fully restoring the S3 service.

Posted Aug 16, 2024 - 16:10 UTC

Resolved

This incident has been resolved.

Posted Aug 07, 2024 - 13:39 UTC

Monitoring

We have identified and resolved the issue affecting the console and bucket/sub-user creation in the system. All systems are operational and we will continue to monitor the services.

Posted Aug 06, 2024 - 13:39 UTC

Update

We continue to investigate this issue. This is also impacting the ability to create buckets and sub-users in the system. Access to current buckets is unaffected.

Posted Aug 06, 2024 - 13:15 UTC

Investigating

Customers attempting to authenticate to the Wasabi Management Console using MFA are still impacted following this morning's Service Incident. We are working to restore this access. S3 connectivity is not impacted by this.

Posted Aug 06, 2024 - 12:09 UTC

This incident affected: US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon), CA-Central-1 (Toronto), EU-Central-1 (Amsterdam), EU-Central-2 (Frankfurt), EU-West-1 (London), EU-West-2 (Paris), AP-Northeast-1 (Tokyo), AP-Northeast-2 (Osaka), AP-Southeast-1 (Singapore), AP-Southeast-2 (Sydney), Wasabi Account Control API, and Wasabi Management Console.