This incident has been resolved.

As a result of a lightning strike, one of our primary data centers in the US central region experienced cooling issues that resulted in an urgent powering down of all servers in the facility. Our global model scheduler was running on general-purpose compute in the impaired data center and the decision was made to re-locate it.

The actions taken resulted in all model instances globally being re-created causing significant degradation of service between 02:08 UTC and 02:24 UTC. Unfortunately, the scheduler isn't currently optimized for rapid model bringup in multiple regions at the same and manual intervention was required which delayed the recovery. By 02:27 UTC errors rates had dropped for ~3% and by 03:34 UTC had returned to standard levels.

We are seeing performance improvement across the impacted models as the team continues to remediate the issue.

We are continuing to work on a fix for this issue.

The team has identified the issue causing varying outage, elevated error rates, and latency across several models. They are working to implement a fix.