Degraded performance in US-EAST-1 and US-EAST-2 regions

Incident Report for Wasabi Technologies

Postmortem

From 2024-05-30 15:36 UTC to 2024-05-31 05:00 UTC, we experienced elevated temperatures in our us-east-1 and us-east-2 regions. This issue was due to a cooling system failure in one of the Iron Mountain Datacenter (IMDC) buildings which hosts our storage sub-system and database servers. This failure caused temperatures to pass a safe operating threshold, causing systems to involuntarily shutdown in order to prevent any damage.

Between 2024-05-30 17:00:00 UTC and 2024-05-31 05:00:00 UTC, our Operations Team worked on each server rack to bring each individual component back up safely, ran integrity checks on the hardware and replaced faulty equipment. At 2024-05-31 05:00:00 UTC, our services were returned to a fully operational status.

Posted Jun 05, 2024 - 20:01 UTC

Resolved

Services in our us-east-1 and us-east-2 regions have been restored and all faults related to the data center cooling issues have been mitigated. For any issues, please reach out to our Support Team at support@wasabi.com

Posted Jun 03, 2024 - 18:53 UTC

Update

Services to our us-east-1 and us-east-2 regions have been restored. Out of an abundance of caution we will continue to monitor the regions throughout the weekend. For any issues, please reach out to our Support Team at support@wasabi.com

Posted May 31, 2024 - 21:15 UTC

Monitoring

Systems have been restored to operational status.
We continue to monitor these services.
If you experience any issue please reach out to our support team.

Posted May 31, 2024 - 03:40 UTC

Update

We are continuing the restoration of the impacted components. A large number of servers have been rebooted & restored and we are working on the remaining servers in both regions. We anticipate the full process to complete between 4-8 hours. We will continue to update our status page.

Posted May 31, 2024 - 01:09 UTC

Update

Recovery operation is currently underway to bring back the impacted systems. This will take between 6 to 12 hours to complete. We will continue to update here as progress is made.

Posted May 30, 2024 - 19:29 UTC

Update

We are seeing a large amount of HTTP 500-level errors being returned to client requests due to this incident. Please check the status page regularly to receive the latest updates.

Posted May 30, 2024 - 16:24 UTC

Identified

Wasabi's Operations Team has been informed that Iron Mountain Datacenters in the us-east-1 and us-east-2 regions are experiencing a cooling issue, impacting the operating temperatures of Wasabi server hardware in those regions.

Posted May 30, 2024 - 15:58 UTC

This incident affected: US-East-1 (N. Virginia), US-East-2 (N. Virginia), and Wasabi Management Console.