Something I've found a lot modern IT architects seem to ignore is "write amplification" or the equivalent effect for reads.
If you have a 1 KB piece of data that you need to send to a customer, ideally that should require less than 1 KB of actual NIC traffic thanks to HTTP compression.
If processing that 1 KB takes more than 1 KB of total NIC traffic within and out of your data centre, the you have some level of amplification.
Now, for writes, this is often unavoidable because redundancy is pretty much mandatory for availability. Whenever there's a transaction, an amplification factor of 2-3x is assumed for replication, mirroring, or whatever.
For reads, good indexing and data structures within a few large boxes (like in the article) can reduce the amplification to just 2-3x as well. The request will likely need to go through a load balancer of some sort, which amplifies it, but that's it.
So if you need to process, say, 10 Gbps of egress traffic, you need a total of something like 30 Gbps at least, but 50 Gbps for availability and handling of peaks.
What happens in places like Twitter is that they go crazy with the microservices. Every service, every load balancer, every firewall, proxy, envoy, NAT, firewall, and gateway adds to the multiplication factor. Typical Kubernetes or similar setups will have a minimum NIC data amplification of 10x on top of the 2-3x required for replication.
Now multiply that by the crazy inefficient JSON-based protocols, the GraphQL, an the other insanity layered on to "modern" development practices.
This is how you end up serving 10 Gbps of egress traffic with terabits of internal communications. This is how Twitter apparently "needs" 24 million vCPUs to host text chat.
Oh, sorry... text chat with the occasional postage-stamp-sized, potato quality static JPG image.
Your point makes sense if you have no idea how Twitter works.
It needs to assemble tweets internally, sort them with an ML model, add in relevant ads and present a single response to the user because end-user latency matters.
And each of these systems eg. ads has their own features, complexities, development lifecycle and scaling requirements. And of course deploying them continuously without downtime. That is how you end up with disparate services and a lot of them for redundancy reasons.
I know you think you’re smarter than everyone at Twitter. But those who really know what they are doing have a lot more respect for the engineers who built this insanity. There are always good intentions.
> I know you think you’re smarter than everyone at Twitter. But those who really know what they are doing have a lot more respect for the engineers who built this insanity. There are always good intentions.
You ignored one possibility - that twitter engineers, or people managing them might be just incompetent and all of that might just be overly complex POS
There is that weird disgusting trend to assume just because company got big that means the tech choices were immaculate, and not everything else there is to successful companies.
You can make perfectly well doing company on totally mediocre product that hit the niche at right time
They wrote Finagle which is a very well written and highly regarded Scala micro-service framework that is used by other companies such as LinkedIn, Soundcloud etc. Likewise Pelikan is an excellent caching service.
We know that Scala is a proven, high-performance, type-safe language that is optimised for concurrency and stability. So it's not like their tech stack is written in Ruby and needs to be re-written in Rust or Go. It's already performant.
If you have a 1 KB piece of data that you need to send to a customer, ideally that should require less than 1 KB of actual NIC traffic thanks to HTTP compression.
If processing that 1 KB takes more than 1 KB of total NIC traffic within and out of your data centre, the you have some level of amplification.
Now, for writes, this is often unavoidable because redundancy is pretty much mandatory for availability. Whenever there's a transaction, an amplification factor of 2-3x is assumed for replication, mirroring, or whatever.
For reads, good indexing and data structures within a few large boxes (like in the article) can reduce the amplification to just 2-3x as well. The request will likely need to go through a load balancer of some sort, which amplifies it, but that's it.
So if you need to process, say, 10 Gbps of egress traffic, you need a total of something like 30 Gbps at least, but 50 Gbps for availability and handling of peaks.
What happens in places like Twitter is that they go crazy with the microservices. Every service, every load balancer, every firewall, proxy, envoy, NAT, firewall, and gateway adds to the multiplication factor. Typical Kubernetes or similar setups will have a minimum NIC data amplification of 10x on top of the 2-3x required for replication.
Now multiply that by the crazy inefficient JSON-based protocols, the GraphQL, an the other insanity layered on to "modern" development practices.
This is how you end up serving 10 Gbps of egress traffic with terabits of internal communications. This is how Twitter apparently "needs" 24 million vCPUs to host text chat.
Oh, sorry... text chat with the occasional postage-stamp-sized, potato quality static JPG image.