You only build the world occasionally, like when cloning or staring a new project, or when upgrading your rustc version. It isn't your development loop.
I do think that dynamic libraries are needed for better plugin support, though.
Unless a shared dependency gets updated, RUSTFLAGS changes, a different feature gets activated in a shared dependency, etc.
If Cargo had something like binary packages, it means they would be opaque to the rest of your project, making them less sensitive to change.
Its also hard to share builds between projects because of the sensitivity to differences.
I do think that dynamic libraries are needed for better plugin support, though.