I understand your point. I host on linode which has Lish (an out-of-bound terminal that gives you command-line access without ssh). I have it turned off normally, but again can be turned on by anyone who gains access to my account with credentials and 2-factor auth. It should take a lot more from an attacker to be able to access your servers.