System Errors in US-EAST-2

Incident Report for Wasabi Technologies

Postmortem

On 23 May 2025 at 23:00 UTC, Wasabi’s us-east-2 (Manassas) vault began seeing a large increase in S3 API traffic, causing regional resources to become strained with excessive internal connections. This excessive, sustained increase in traffic to the region was outside of normal operating conditions. At 08:12 UTC on 24 May 2025, this high resource usage began to impact the vault, causing an increase in 5xx errors for API calls to S3 buckets. Upon investigation from our Operations Team, it was found that a small number of accounts were responsible for consuming a large portion of available resources within the region, causing the increase in HTTP 5XX error responses for other customers. Our Operations Team took action by limiting the available connections to these accounts, which allowed for the overall active connections to the region to drop substantially. By 08:32 UTC on 24 May 2025, the region was back to normal operation.

Posted Jun 03, 2025 - 03:46 UTC

Resolved

This incident has been resolved. Please contact support@wasabi.com if you continue to see errors related to this incident.
Posted May 24, 2025 - 16:29 UTC

Monitoring

We have made changes, traffic is operating normally, and we are monitoring the systems.
Posted May 24, 2025 - 13:10 UTC

Identified

The issue has been identified, and we are making appropriate changes to the system.
Posted May 24, 2025 - 12:46 UTC

Investigating

We are currently experiencing system errors in our US-EAST-2 region. Customers may experience elevated HTTP 5XX error responses when interacting with their Wasabi bucket(s). We will update this page as we have more information.
Posted May 24, 2025 - 11:22 UTC
This incident affected: US-East-2 (N. Virginia).