That's what the python implementation did. Except the lightweight binary to interface with it was also in python. But that still had a massive win, not having to load all the modules every time.
I actually rewrote the interfacing binary in rust about two years ago, that would talk to the python server. The python equivalent would start the server if it wasn't already running, but rust, back then, didn't support spawning a subprocess and keeping it alive after the parent process quits, so it was never deployed as a replacement. I filed an issue about that, and iirc, that was fixed, but I never came back to finish and deploy the lightweight rust client. I probably should have.