Yes technically it could work, but I cannot ask the users to use SSH and configure VNC. The force of team viewer is that you download it, open it, and give number over the phone and it works.
But for remote control, it stops after every few minutes, asking the "controlled" user to click on a button to continue. Not so practical in a few situations.
It always happens for me when accessing a Linux machine remotely. I can't find a screenshot now, so the next time I do it, I'll take one. It seems to be a security measure, to prevent someone from sharing remote access and then forgetting about it afterwards, but it makes for terrible usability.