A lot of things work that way because it’s obvious to everyone except the container runtimes on Linux.

The container runtimes have to cope with Dockerfiles and similar, which know nothing about packages. To get the kind of granularity you want here, you have to do actual packaging work, which is the thing Docker sold everyone on avoiding.

If you are willing to do that kind of packaging work you can get the best of both worlds today with Nix or Guix. But containers are attractive because you can chuck whatever pathological build process your developers have evolved over the decades into a Containerfile and it'll mostly work.