If I were you, I would follow up on your thoughts with something like:

"I don't think it's appropriate for us to make remote or in-office location related to salary. You're hiring me to get things done. I incur higher costs by maintaining a higher office. What recompense do you offer for costs of running a remote office since your office costs are deferred?"

Or maybe softer, but that should take the idea off the table for them.