Before the tire fire that is rvm became "standard", it was normal to just download the version you needed and build it from source. There's nothing forcing us to use crappy, obscure tools like rvm and rbenv. And I agree with you on all points.
It's a total shitshow. But since ruby manages everything with environment variables, you need some kind of tooling to manage multiple versions. Horrendous.
Ruby should just standardize on a way to point projects to rubies. Instead of a zillion environment variables, just a shebang. It's how everything else works.