"The main problem with the M1 for me is the GPU and other devices around it"

That's a separate issue than the ISA extensions, which apply to the ARM CPU.