The bottom line is you're not putting money where your mouth is. Real rugged devices can't have top notch performance because waterproofing and expanding wider operating temperature require insulation and therefore performance reduction but that's not important.
The very core of the problem is you - not personally but the vast majority would-be rugged phone buyers - just don't buy rugged phones, nor take it outdoors. People who'd demand rugged phones would just buy the latest and greatest iPhone, maybe with a case with reward points, and that covers almost every single use cases.
If there had been demand at all, the level of performance possible in a ruggedized phone will be the benchmark, and current high end will be considered over the top models with compromised ruggedness, but the reality isn't working that way at all.
And note that it comes down to what threat level it actually faces. I take my perfectly ordinary smartphone into the wilderness. It's never going to tumble down a rock face both because you'll never find me on one but because I have it lanyarded to me. It's fallen a few times, but the lanyard has kept it from hitting the ground.
You have to balance the cost of it being rugged vs the expected chance of the ruggedness keeping it from being damaged. And for most people the tradeoff isn't worth it.
That's the problem with these "niche" phones. Another example is Fairphone.
I don't necessarily have a problem with spending more, my problem is the fact that they compromise on things they could not compromise.
See Framework, they did it right.