Network issues affecting all regions

Incident Report for Wasabi Technologies

Postmortem

On 30 November 2024 from 2024-11-30 00:17 UTC to approximately 2024-11-30 03:00 UTC, Wasabi experienced an issue where client connection attempts to Wasabi Cloud Storage and the Web Console were impacted across all storage regions, resulting in all API calls to be returned as HTTP 5XX errors to clients. The cause behind this service degradation was due to an error within the internal messaging queue service responsible for taking client requests and routing them to our global database cluster. The internal messaging queue service failed to appropriately route these client requests across all nodes.

Wasabi’s Engineering and Operations teams was able to mitigate this issue by manually configuring internal servers to route requests across multiple database instances, allowing the system to recover and respond to requests appropriately. Once this action was taken, which restored service, the teams then worked to correct the root cause by working to recover the internal messaging queue and resume the automated task of proper client request handling.

Posted Dec 04, 2024 - 14:09 UTC

Resolved

This incident has been resolved.

Posted Nov 30, 2024 - 04:46 UTC

Update

The system has been restored to fully operational in all regions. We will populate the Postmortem section of this incident with more complete details as soon as possible.

Posted Nov 30, 2024 - 04:45 UTC

Monitoring

We have isolated the issue and resolution is underway. We expect all regions except us-east-1 to be successful. There is still some level of error responses being seen in Ashburn and we continue to work on that.
We will continue to monitor all regions and update their status as well as that of us-east-1 as we continue to make progress.

Posted Nov 30, 2024 - 03:43 UTC

Update

We continue to investigate the issue. Some traffic continues to receive errors.

Posted Nov 30, 2024 - 03:08 UTC

Investigating

We are currently investigating reported network errors across all regions. Access to both Console and S3 services may return errors. We will update this page as we have more information.

Posted Nov 30, 2024 - 00:20 UTC

This incident affected: US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon), CA-Central-1 (Toronto), EU-Central-1 (Amsterdam), EU-Central-2 (Frankfurt), EU-West-1 (London), EU-West-2 (Paris), EU-West-3 (London), EU-South-1 (Milan), AP-Northeast-1 (Tokyo), AP-Northeast-2 (Osaka), AP-Southeast-1 (Singapore), AP-Southeast-2 (Sydney), Wasabi Account Control API, Wasabi Account Control Manager Console, and Wasabi Management Console.