I think this part of the cascade shadow map generation could be improved. Luckily the stuff closest to the camera looks correct, but distant stuff does look off until you get closer and higher resolution pops in.

In the future I'll probably do something similar to what you mentioned and have the cascades use the LODs selected for the main view.