As an outsider to network topology that's fascinating to understand. Thanks for explaining.
If you don't mind going further, how does an ISP (or anti-DDoS services) mitigate something like this without incurring even more overhead to examine the traffic & not route it onwards? Just a flat block against the sources, or something more sophisticated?
ISP mitigation is usually simple. Figure out what the destination of the attack traffic is and null route[1] that, pushing those null routes upstream where possible. If the destination IPs for the abuse are too diffuse and/or the upstream won't do it on their routers where they presumably have more capacity to drop than to forward to you, you're still limited by your connection bandwidth on the connections the abuse is flowing. Depending on the use case, sometimes you can do pretty well by cycling your legitimate traffic across a large block of IPs faster than the abusers can cycle their abuse traffic, but it depends on relationships and APIs to control null routing of your upstreams. More sophisticated things like only block traffic to/from port X, or throttling rather than blocking aren't generally available (but would often be super helpful if they were).
Anti-DDoS services are quite a bit different, they generally have obscene bandwidth, and BGP advertise your attacked IPs, so they can get the traffic, then they "scrub" the traffic and provide it back to you over a tunnel (or something). Scrubbing can be relatively simple stuff like rate limiting UDP and especially fragmented UDP (which is usually stuff coming from reflection attacks) or more complex stuff like looking at TCP options to determine good vs bad traffic and only letting good traffic through, or tracking TCP SYNs and only letting retries through, plenty of other stuff with TCP state machines (or half state machines, if they're only seeing the incoming traffic and not the outgoing traffic). Oh, and they usually charge you gobs of money.
It's only simple if you don't care that the service behind that IP is down. I would only use it as a last resort if there are no others tools available or if i was a consumer ISP. Any competent ISP with business customers, hosting or otherwise, wouldn't be doing this.
Null routing is often (but not always) used in the scenario that the service behind the IP is already down, but null routing will improve service to everyone else, so it's an acceptable casualty.
I was (working for) a major customer of a competent hosting service and they'd do null routes (automated) on our service IPs when we got attacked. Of course we complained, but it was that or have automation put us behind a traffic 'scrubber' that caused more problems than null routing. In the end, what made it right for us was when they got the traffic levels and sampling intervals set properly; only null route if incoming traffic is above line rate for the server, and initially null route for a short time (5 minutes, I think) before resampling and escalating the time. Most attacks were short, so the default of 6 hours was excessive.
I mean, yes, but a) they didn't have the equipment that could do that at the rate required, b) they certainly couldn't ask their upstreams to do that and at least for some of the attacks, the attack traffic added enough traffic to go over capacity of some of their links; having the upstream null route it was possible and solved the congestion.
Haven't done much traditional networking for a few years so my knowledge is not that current.
They use bgp flowspec to send out filter lists using the bgp protocol to routers, essentially a long list of ACL rules with srcip dstip dstport. These routers can filter any amount of traffic it gets in hardware, so as long as your upstream (edge transit or peering links) are not full you won't even notice the ddos.
But you need something to analyse traffic that can understand which traffic is ddos and what is normal. We used devices from Arbor which are basically regular x86 servers. These devices also received netflow so whenever a ddos target received a lot of weird traffic (lots of DNS for example) it was configured to redirect all traffic for that IP to itself where it could gain a better understanding than what you can from netflow and either filtered it locally or sent out flowspec updates to filter it on routers. You could also enable this manually for non-volumetric ddos and do some filtering, but it's hard to do if it's encrypted and if they are abusing some http call that is just expensive to handle for the servers.
If you don't mind going further, how does an ISP (or anti-DDoS services) mitigate something like this without incurring even more overhead to examine the traffic & not route it onwards? Just a flat block against the sources, or something more sophisticated?