Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

We do run an anycast CDN network, but there's a lot of limitations on BGP routing compared to CDN based traffic direction.

We can only route BGP requests to hardware we control, whereas we can add PoPs in all the major cloud providers on our DNS based network. We can then use tools like Cedexis or DYNs internet intelligence to identify where the different cloud providers have the best networking and peering agreements and piggy back on that + their DDoS mitigation. This means we get a combination of all the best AWS/Google Cloud/Rackspace/DO, etc, etc has to offer in that aspect.

On the DNS based traffic director we can also do very quick traffic decisions (20s TTL, instant changes) whereas on our BGP routed anycast IP we have to be more conservative and force 10 minute intervals between any up/down changes for a PoP.



I did GeoDNS + Unicast IPs for a while. I had a really rough time making it work, and we ended up building our own anycast network (https://status.neocities.org)

Aside from the root domain issues (and less options for market-priced bandwidth), "GeoDNS + Cloud" pushes your traffic into someone else's ASN, which means complaints end up being sent to them, and your hosting is effectively governed not just by one, but by two different ToSes.

This isn't a big deal for a couple thousand sites (unless they're huge), but once you start getting into the hundreds of thousands, you'll see a significant spike in issues (phishing, malware, spam, DMCA, legal threats, etc.) that get sent to whomever owns that IP address. After getting too many of these complaints, those other providers can decide you're just not worth the effort and boot you off their servers.

Crazy hypothesis? Sounds like it would be, but it happens: https://twitter.com/surge_sh/status/685164708861624325. DO did the same thing to us when we tried to use them for part of our CDN early on. After that, I tried three other cloud services that either did the same thing or threatened to do the same thing (to say nothing about the ridiculously overpriced bandwidth).

The choice we were left with: Get our own AS, or die. Mind you, this was over < 30 abuse reports per month, not thousands. Most of these providers are designed for a single company or a wordpress blog, they're not designed (and not really equipped) for usage as infrastructure for a web hosting provider with hundreds of thousands (or millions) of customers.

Building out the anycast CDN was a "drinking from the firehose" experience and had some upfront costs I would have rather not paid, but it solved this existential problem for us permanently, and probably saved our life. From experience, I do think you'll have to do this eventually (or at least do GeoDNS + unicast with your own IPs and AS).


Have you written up your experience with building out the anycast CDN? That would be extremely interesting!


I'd be interested in reading that too




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: