Skip the conversation entirely?

I don't really have this problem, but my co-workers often put several-hour meetings on their calendar with the meeting name and participants hidden. Seems to work to prevent others from inviting them to meetings during that time.