If I understand you correctly, you're advocating for always supporting a minimalist COM-like object format. The object code needs to be in some file, and formatted in some way.
That's fine, but it's different from being object format independent.
What does "pure binary" even mean, though? I think you mean a bare-bones COM-like format where the binary gets loaded at an ABI-determined static address in memory, the instruction pointer is set to the first byte of the binary, and you're off to the races.
This "pure binary" is still an object format, and still depends (at minimum) on the format's convention for the address at which the binary is loaded. Note that this is the case even when writing firmware ROMs, where the architecture determines where the boot ROM gets mapped in memory.
Old compilers used to output pure binary by default; newer ones will usually choose a container format (ELF, EXE, etc.) by default, and may not have the pure binary option available.
That's fine, but it's different from being object format independent.