Investigating System Outage on US-West-1
Incident Report for Wasabi Technologies
Postmortem

On 2 November 2023, a major power issue was encountered in the us-west-1 region infrastructure. Multiple utility power sources failed and power was reverted to generators to maintain services. Several hours later, prior to the restoration of local power, the redundant generators failed and Wasabi services (and services from other SaaS and IaaS tenants at this location) were impacted. Our facility service providers worked with local power companies to first recover the generator power and then restore local power. Once power was restored, we were able to restore services. The providers continue their investigations to isolate the cause of the initial power failures, but faults have been identified in the generator system and repairs are scheduled there to address that issue.

Posted Nov 07, 2023 - 18:26 UTC

Resolved
This incident has been resolved.
Posted Nov 03, 2023 - 13:42 UTC
Update
Our us-west-1 storage region is fully operational. We will continue to monitor the region.
Posted Nov 03, 2023 - 00:08 UTC
Monitoring
Power to Wasabi’s us-west-1 storage region has been restored. Most components have been returned to service which has helped remediate the impact. We are currently working to restore the remaining affected services and return to fully operational status.
Posted Nov 02, 2023 - 20:17 UTC
Update
16:30 UTC 2023-11-02 - We are continuing to work with the US-West-1 data center to ensure power stability, while simultaneously continuing efforts to restore the S3 service to be fully operational. We expect some impact to customer traffic during this time.
Posted Nov 02, 2023 - 16:44 UTC
Identified
We have restored power to the US-West-1 data center and are working on restoring the S3 service.
Posted Nov 02, 2023 - 12:56 UTC
Update
We are continuing to investigate this issue.
Posted Nov 02, 2023 - 12:46 UTC
Investigating
We have noticed an issue with electrical feed into the us-west-1 region, we are working to have this restored now.
Posted Nov 02, 2023 - 12:21 UTC
This incident affected: US-West-1 (Oregon).