Edit: it’s worth expanding on the data centre costs issues - if it’s fair to say we have stalled on “free”speed ups for standard software (ie C/Linux) - that is clock speeds more or less stopped and we get more 64 bit cores but each core is more or less no faster (hand wavy), then the amount of clients that can be packed into a data centre stays the same - you are renting out CPUs in a Docker VM - that’s basically one per Core(#). And while wafer scale gives you 900,000 cores in a 1u server, normal CPUs gives you what 64? 128?
Suddenly your cost for building a new data centre is something like twice the cost of the previous one (cold gets more expensive etc) and yet you only sell same amount of space. It’s not an attractive business in first place.
This was the push for lambda architecture etc - depending on usage you could have hundreds of people buying the same core. I would have put a lot of cash into making something that spins up docker instances so fast it’s like lambda - and guess what fly.io does?
I think fly.io’s obsession with engineering led them down a path of positive capital usage while AWS focused on rolling out new products on a tougher capital process.
Anyway - AI is the only thing that dense multi core data centres can use that packs in many many users compared to docker per core.
Unless we all learn how to think and code in parallel, we are leaving a huge amount of hardware gains on the table. And those gains are going to 100x in the next ten years and my bet is my janky-ass software will still not be able to use it - there will be a bifurcation of specialist software engineers who work in domains and with tooling that is embarrassingly parallel, and the rest of us will be on fly.io :-)
(#) ok so maybe 3 or 4 docker per core, with hyper visor doling out time slots, but much more than that and performance is a dog, and so the number of “virtual CPUs” you can sell is a limited number and creeps up despite hardware leaping ahead … the point I am making
Web applications and APIs for mobile apps are embarrassingly parallel, but many modern web languages and frameworks went back to the stone age on parallelism.
Ancient Java servlets back in the early 2000s were more suitable for performance on current gen hardware than modern NodeJS, Python etc...
It’s worth thinking about how to architect some of this in the new hardware.
To go extreme, wafer scale silicon - 900,000 8 bit cores, 120GB sRam. Hand wave on the 8bit for now and just think how to handle facebook or e-commerce. A static site is simple if it fits inside 120GB, but facebook is low write high read (Inassume) - but fetching from / writing to memory - making that parallel means rewriting a lot of things .. and e-commerce - suddenly ACID does not parallelise easily.
I mean this all seems doable with fairly fundamental changes to architecture memory concepts and … rdbms is challenging.
But I think we are way past the idea that a web framework author can just fix it - this is deeper - a new OS a new compiler and so on - but I could be wrong.
Suddenly your cost for building a new data centre is something like twice the cost of the previous one (cold gets more expensive etc) and yet you only sell same amount of space. It’s not an attractive business in first place.
This was the push for lambda architecture etc - depending on usage you could have hundreds of people buying the same core. I would have put a lot of cash into making something that spins up docker instances so fast it’s like lambda - and guess what fly.io does?
I think fly.io’s obsession with engineering led them down a path of positive capital usage while AWS focused on rolling out new products on a tougher capital process.
Anyway - AI is the only thing that dense multi core data centres can use that packs in many many users compared to docker per core.
Unless we all learn how to think and code in parallel, we are leaving a huge amount of hardware gains on the table. And those gains are going to 100x in the next ten years and my bet is my janky-ass software will still not be able to use it - there will be a bifurcation of specialist software engineers who work in domains and with tooling that is embarrassingly parallel, and the rest of us will be on fly.io :-)
(#) ok so maybe 3 or 4 docker per core, with hyper visor doling out time slots, but much more than that and performance is a dog, and so the number of “virtual CPUs” you can sell is a limited number and creeps up despite hardware leaping ahead … the point I am making