They're using the land/housing that could otherwise be used by someone who actually does live there. Just because you're choosing not to spend your time there shouldn't matter.