Yeah that works if you are not dealing with open source. If you are dealing with open source, though, it really won't save you that much trouble, if anything it will just lead to unnecessarily hostile interactions. You're not really obligated to fix any specific issues that people report, but shrugging and saying "Your problem." is just non-productive and harms valuable downstreams like Linux distributions. Especially when a lot of new failures actually do indicate bugs and portability issues.
It doesn't even work outside of open source. I am running a prerelease toolchain almost all the time on my computer. If the project at work turns on -Werror, I immediately turn it off and store away the change. Of course this means that I send in code fixes for things that don't reproduce on other people's machines yet, but I literally never receive pushback for this.
Supporting every Linux distribution and their small differences isn't free, and Linux distributions shipping things you haven't tested directly is also a way for users to get bitten by bugs or bad interactions, which they will then report to you directly anyway so you're responsible for it. It's complicated. It's happened plenty of times where e.g. I've run into an obscure and bad bug caused by a packaging issue, or a downstream library that wasn't tested -- or there's a developer who has to get involved with a specific distro team to solve bugs their users are reporting directly to them but that they can't reproduce or pinpoint, because the distro is different from their own environment. Sometimes these point out serious issues, but other times it can be a huge squeeze to only get a little juice.
For some things the tradeoffs are much less clear, open-source or not e.g. a complex multi-platform GUI application. If you're going to ship a Flatpak to Linux users for example, then the utility of allowing any random build environments is not so clear; users will be running your produced binaries anyway. These are the minority of cases, though. (No, maybe not every user wants a Flatpak, but the developers also need to make decisions that balance many needs, and not everything will be perfect.)
Half of the problem, of course, is C and C++'s lack of consistent build environments/build systems/build tooling, but that's a conversation for another day.
That said, I generally agree with you that if you want to be a Good Citizen in the general realm of open-source C and C++ code, you should not use -Werror by default, and you should try (to whatever reasonable extent) to allow and support dependencies your users have. And try to support sanitizers, custom CFLAGS/CXXFLAGS, allow PREFIX and DESTDIR installation options, obey the FHS, etc etc. A lot of things have consolidated in the Linux world over the years, so this isn't as bad as it used to be -- and sometimes really does find legitimate issues in your code, or even issues in other projects.
Again, you don't have to fix bugs that are reported, but treating it as invalid to use any compiler versions except for the exact ones that you use is just counterproductive.
The "utility" of allowing "any random build environment" is that those random build environments are the ones that exist on your user's computers, and absent a particularly good reason why it shouldn't work (like, your compiler is too old, or literally broken,) for the most part it should, and usually, it's not even that hard to make it work. Adopting practices like defaulting -Werror -Wall on and closing bugs as WONTFIX INVALID because it's not any of the blessed toolchains gains you... not sure. I guess piece of mind from having less open issues and one less flag in your CI? But it is sure to be very annoying to users who have fairly standard setups and are trying to build your software; it's pretty standard behavior to report your build failures upstream, because again, usually it does actually signal something wrong somewhere.
Developers are free to do whatever they want when releasing open source code. That doesn't mean that what they are doing is good or makes any sense. There are plenty of perfectly legal things that are utterly stupid to do, like that utterly bizarre spat between Home Assistant and NixOS.
C++ is super annoying in this way. Many other languages (e.g Rust) only have one compiler and good portability out of the box which completely avoids this problem. And other ecosystems that do have multiple implementation (e.g. JavaScript) seem to have much better compatibility/interop such that it's not typically a problem you have to spend much if any time on in practice.
I'm curious what sort of CPUs and OSes do those languages run on. C++ runs on all sorts of obscure real time OSes, all the standard mainstream ones as well as on embedded equipment and various CPUs, but a lot of that is possible because of the variety of compilers.
