> Like what exactly did they gain by making a weird non-PCIe NVMe situation?
When Intel did exactly that, there was a clear plausible chain of decisions leading to that madness. I have no clue what may have led Apple in this direction, but the excuses probably aren't any more pathetic than trying to explain why Intel has shipped two mutually-incompatible "solutions" for preventing NVMe drives from working out of the box with unmodified Windows.
VMD is weird, but at least it doesn't require big intrusive changes like decoupling your NVMe driver from PCIe. It's more like a weird special PCI-PCI bridge. Only needs a little extra driver: https://reviews.freebsd.org/D21383
VMD isn't the only method Intel has used to mess with how NVMe works. Their consumer chipsets going back at least to Kaby Lake had an even weirder "feature" that hid NVMe devices from PCIe enumeration and made them only accessible through proprietary interfaces on the chipset's SATA controller. Intel had to start using VMD on consumer platforms instead when AMD forced them to start providing more PCIe lanes from the CPU.
When Intel did exactly that, there was a clear plausible chain of decisions leading to that madness. I have no clue what may have led Apple in this direction, but the excuses probably aren't any more pathetic than trying to explain why Intel has shipped two mutually-incompatible "solutions" for preventing NVMe drives from working out of the box with unmodified Windows.