From 2024-06-19 00:57 UTC to 2024-06-19 04:36 UTC, we experienced issues in all Wasabi regions affecting S3, Wasabi Account Control (WAC) API, and Console services. Our WAC API and Console services were affected between 00:57 UTC - 02:45 UTC and 03:55 UTC - 04:36 UTC, while our S3 service was affected between 01:55 UT - 02:45 UTC. 

At 00:57 UTC it was noticed by our Operations Team that there were connection issues between our Console and WAC API services and our global database with no impact to other (S3) services. While debugging these connection issues, it was noted at 1:55 UTC that S3 services were now also affected. After further investigation from the Operations Team, it was seen that our queuing service within the global database had failed, resulting in these services unable to communicate with the database. 

To correct this issue, Operations had to restart both the global database instance, as well as the queuing service to restore connections for the affected services. Once both the database and queuing service had re-initialized at 2:45 UTC, services were restored. 

At 03:55 UTC, it was noticed that the Console and WAC API were beginning to exhibit the same symptoms as before with a loss of connectivity to the database. Upon further investigation from the Operations Team, it was noted that the queuing service had run low on resources. Operations increased available resources to the service and at 04:10 UTC the services were brought back online. At 04:36 UTC it was confirmed that all services were operational.