The IIHS has stats on this and it does seem that larger vehicles have lower average death rates[1].
In a world of only smaller sized cars, having a larger car would likely be less safe for the reasons you listed. However, with large cars being so popular, it would make sense that being in a smaller car is less safe as you are more likely to be in a collision with a vehicle that has more mass than yours—not to mention you have less room for crumple zones and the larger vehicle would hit you at a higher point on your vehicles, which is known to be more lethal.
In a world of only smaller sized cars, having a larger car would likely be less safe for the reasons you listed. However, with large cars being so popular, it would make sense that being in a smaller car is less safe as you are more likely to be in a collision with a vehicle that has more mass than yours—not to mention you have less room for crumple zones and the larger vehicle would hit you at a higher point on your vehicles, which is known to be more lethal.
[1]: https://www.iihs.org/topics/fatality-statistics