I think the idea (with which i fully agree, btw) is to avoid at all technically possible costs to make backwards incompatible changes in the first place (and when they happen they should be considered as nothing more than bugs to be fixed - ie. restore the backwards compatibility) and the semantics of semver are trying to represent some state that shouldn't be representable at all.

Think of it as a reference in C++ that is null... it can technically happen (in the sense that all bytes that store the reference are, e.g., zero), but the language doesn't really allow for it and if it does happen it is a bug.