Google: Quota regulation for API access led to cloud outage

Following the global outage of Google's cloud infrastructure on Friday night, the company has provided initial information on the cause. According to initial findings, a faulty update to the quota system for API accesses triggered the disruption, Google announced.

As a result of the error, an increased number of 503 errors (“Service Unavailable”) were reported for API calls. Numerous Google services were affected, but the error also caused massive problems for other service providers.

Three hours

The first outages were reported shortly before 8pm (CET) on Thursday evening. According to Google, the disruption began at 7:49 pm and lasted a total of three hours. Once the cause had been found, Google suspended the faulty quota checks.

Subsequently, services in most regions returned to normal within two hours, Google added. Only in the US Central region was a database for quota management so overloaded that it took much longer to fix. Services have since been largely restored.

“We deeply regret the impact the outage and service interruption had on all our users and their customers,” writes Google. “Businesses large and small trust Google Cloud with their workloads, and we will do better.” The company plans to provide a full report “in the coming days”.

Other services affected

The outage at Google had a serious impact on numerous other services. In addition to Cloudflare, Spotify, YouTube, Discord and other providers also reported disruptions.

Meta also discovered in autumn 2021 just how quickly a small error can paralyze a global service. After an incorrect command during maintenance work, the connections between Facebook's data centers were cut worldwide. The sites were also no longer accessible from outside. As a result, the world was without Facebook, Instagram, and WhatsApp.

(vbr)