I never got RVM working with fish; unsurprising, seeing as it’s 20k lines of bash. rbenv works well though (with one additional conf line), and chruby was working on support last time I checked.
I started using RVM when it was the only (possibly well-known?) game in town, and never switched until I started using fish and found that it didn’t work.