Last time I checked, it was completely possible to install Virtualbox and normal Docker CLI on a Mac, and everything worked well (for me), without Docker Desktop at all.
But that last time was on Intel Macs; maybe the situation is completely different on ARM.
It doesn't work on M1 last I checked. And anyway even if they updated Virtualbox for ARM, Oracle is predatory and sends nasty emails about paying up if they detect anyone in your company uses it.
I remember that the open-source Virtualbox part is free from any nagware. The closed-source parts are those needed for desktop integration (good screen, clipboard, file sharing, etc), but they are not needed for running a Linux VM with Docker (or some other container runtime BTW).
But that last time was on Intel Macs; maybe the situation is completely different on ARM.