We are currently investigating system errors in US-EAST-1

Incident Report for Wasabi Technologies

Postmortem

On Sunday, January 21st 2024, at approximately 12:00 UTC a hardware failure in our US-EAST-1 datacenter resulted in two database nodes within the region to reconnect to our cluster while in a non-functional state. The introduction of these nodes to the database cluster within the region caused the database to become unhealthy, resulting in an inability for this cluster to service incoming requests from clients due to an exponential increase in unhealthy connections. By 16:30 UTC our Engineering Team had identified the cause of these unhealthy connections and by 19:00 UTC had prepared a fix to restore this fault. By 20:15 UTC the fix was fully deployed across the cluster, bringing the connection state back to healthy and available for incoming client requests.

Posted Jan 30, 2024 - 14:46 UTC

Resolved

This incident is now resolved. Please reach out to support@wasabi.com if you continue to see any errors related to this incident.

Posted Jan 21, 2024 - 20:57 UTC

Identified

We have identified the issue and made changes to our Database to mitigate the system errors. We are continuing to monitor the system.

Posted Jan 21, 2024 - 20:39 UTC

Update

Our Operations team continues to work with the S3 team to isolate the problem, we will update this page with more information soon.

Posted Jan 21, 2024 - 18:38 UTC

Investigating

We are currently investigating system errors in the US-EAST-1 region and our Operations team is investigating.

Posted Jan 21, 2024 - 16:11 UTC

This incident affected: US-East-1 (N. Virginia).