Can’t they just say the same right back to you?

I agree with your position. It’s just worth noting that you do indeed have to be willing to walk, potentially permanently.