On 17 March 2025 from 10:34 UTC to approximately 10:50 UTC, Wasabi services in our Ashburn (us-east-1), Manassas (us-east-2), Hillsboro (us-west-1), Plano (us-central-1), Amsterdam (eu-central-1), Tokyo (ap-northeast-1), Osaka (ap-northeast-2), London (eu-west-1), and Frankfurt (eu-central-2) regions were unavailable due to an issue which impacted the ability for internal services to communicate effectively within all Wasabi services. Wasabi Engineering and Operations teams were notified by our automated monitoring systems at 10:34 UTC of an issue with a spike in connection between internal service worker nodes. This spike led to the inability to open new connections within the services.
Wasabi’s Operations Team noted that automatic restarts of the services did not clear the issue, prompting manual intervention to perform a manual restart of internal service nodes, which corrected the issue and allowed for client connections to resume successfully. By 10:50 UTC, all internal services were restarted and all services returned to normal operating conditions.