Dependencies of these big providers like Google, Microsoft, Cloudflare are increasing which results to failure on a wide scale even if one fails. Distribution is the key.
Well for the vast majority of simple apps you're better off failing when everybody else is. People will blame it on you less. When your alternative solution fails and everything else seems to be up the blame will fall on you.
I always prefer to have a backup solution which could at least crawl during these situation if not able to walk. I see many SaaS relying only on google/twitter/fb auth but they need to understand that having own system too won't harm them much.
Google could probably do a better job here and not put so many services on the same pool of L7 devices. Separate pools with smaller groupings would reduce the blast radius.
(Googler, opinion is my own, I know nothing about this specific outage).
Google has LOTS of internal routing systems. BGP is about anouncing what IPs a given network can handle, which is not the case here.
Before hitting application level routing, I believe you hit the Maglev[0]. Seems unlikely this was the cause, as it would likely take down all services.
One of the first application layers balancers you hit that is well known is the GFE[1][2]. This is similar to an HTTP reverse proxy, but Google made. I could definitely see this as the cause.
Does that match the list of reported stuff that was down? It appeared to hit a wide range of services. Gmail, Analytics, GKE, Google Keep, Meet, YouTube, GCE buckets, Sheets, Docs, Calendar, Stadia, Firebase, Voice, Music, Nest. From the thread: https://news.ycombinator.com/item?id=24585478
Neither Maglev nor GFE are usually tied to a specific service nowadays, so it could still be either of them. Way back when, some teams or services such as Checkout had to run their own private pool of GFEs. Given Urs' mention of backends, I am slightly inclined toward GFE.
Traffic entering Google's network hits a bunch of front ends that route traffic to the relevenat back ends. I'd guess it's those application-level front ends that were having trouble, rather than anything network-level like BGP.
There's a huge """secret""" Google data center in Council Bluffs, Iowa that appears to be in the finishing phases of completion. I talked yesterday to a union worker who is moving to Des Moines to work on a new Microsoft data center there tonight, it appears that work is drying up at this data center here and a lot of the travelling blue collar folk are leaving this area.
I wonder if this data center coming apparently partially online is a part of the problem?
Also, after this he is likely to work on an Amazon fulfillment center next year - impressed by all the (albeit temporary) blue collar jobs created by FAANG at the moment!
I should have been less esoteric, but yes this one. From what I can tell a gaggle of Google employees have taken over control of the building and I'd assume this absolute unit is in the process of coming online. Sure there's a Google sign visible from the private road leading up to it but you'd still have to be a nosy local (or apparently an all-knowing HN reader) to know the exact location.
One building of it opened in 2013 (and another in 2016 AFAIK), but it is currently still under construction (in the Google Maps view, the entire construction site south of the completed buildings are also slated to be Google DCs).
