On 07 May 2024 at 15:40 UTC, Wasabi had an issue in the US-EAST-1 (Ashburn) vault which resulted in severely degraded performance in the region. This impacted customer’s ability to send and retrieve data to/from their buckets in the region.  

At around 15:40 UTC, the Wasabi Operations team was deploying updates in the US-EAST-1 region which caused our queueing service to fail to accept new connections due to an overloaded memory issue. The queuing service failing to accept new connections caused issues with communication to our global database cache, which resulted in an inability to serve customer requests. To mitigate this issue, our Operations Team found it necessary to restart the global database cache and queuing service. Once the queuing service was restarted, it was allotted additional memory resources to avoid another overload scenario while the deployment was completed. The restarting of both services resulted in connections being restored for the vault. 

At 16:02 UTC, 07 May 2024, all services were restored to the US-EAST-1 region.