On x86, that's the reality. I recently went Intel CPU & AMD GPU to AMD CPU & Nvidia GPU and just booted straight into my old Windows system, no problem at all.
But on anything ARM based? The whole ecosystem doesn't give a fuck. Everything is hyperspecific for one platform exactly. At one point, the Linux kernel had thousands of board source files that specified exactly what hardware your cursed ARM board had. Then they invented device tree but frankly the situation has barely changed, don't expect to ever build a generic ARM image for anything.
You are describing specifically "embedded junk" (which includes the M1, sorry Apple) not literally "anything ARM based". SBSA-compliant platforms boot with UEFI+ACPI, using fully generic DSDT descriptions for PCIe (ECAM) and USB (XHCI). Speaking of GPUs, quite common for them to have QEMU in firmware to run the x86-compiled EFI GOP driver straight from the card's ROM and get video output before the OS.
It's cheaper to use custom ARM boards than it is to build hardware that is SBSA-compliant, so consumers can expect that most, if not all, of their ARM devices are not SBSA-compliant.
But on anything ARM based? The whole ecosystem doesn't give a fuck. Everything is hyperspecific for one platform exactly. At one point, the Linux kernel had thousands of board source files that specified exactly what hardware your cursed ARM board had. Then they invented device tree but frankly the situation has barely changed, don't expect to ever build a generic ARM image for anything.