System Errors in US-WEST-1 Region

Incident Report for Wasabi Technologies

Postmortem

From 2024-09-25 13:49 UTC to 2024-09-25 13:54 UTC, an incident occurred in the Hillsboro region, resulting in performance degradation for customers and impacting their ability to make S3 API calls against buckets in the region.

During an upgrade to the region, the Operations team encountered an issue where a misconfiguration caused a service to falsely report as ready for operation. This resulted in service readiness failures and triggered failures within the region. To address the situation, the Operations Team corrected the misconfiguration and restarted the affected service.

By 13:54 UTC, the S3 service in the US-WEST-1 region was fully restored, and normal operations resumed.

Posted Sep 27, 2024 - 18:05 UTC

Resolved

This incident is now resolved. Please reach out to support@wasabi.com if you continue to see any errors related to this incident.

Posted Sep 25, 2024 - 15:22 UTC

Investigating

We are continuing to investigate this issue.

Posted Sep 25, 2024 - 15:20 UTC

Monitoring

Our Operations Team has corrected the issue, and the US-WEST-1 region is back in an operational state. We will continue to monitor the region.

Posted Sep 25, 2024 - 14:04 UTC

Identified

Our Operations Team has identified the issue and is working to correct the system errors.

Posted Sep 25, 2024 - 13:56 UTC

Update

We are continuing to investigate this issue.

Posted Sep 25, 2024 - 13:52 UTC

Investigating

We are currently investigating system errors occurring in our US-WEST-1 region.

Posted Sep 25, 2024 - 13:51 UTC

This incident affected: US-West-1 (Oregon).