Most white, straight men aren't most people. Heck, all straight, white men aren't most people. Heck, not even all men are most people.

It's not very nice to immediately be racist when someone says something that you could observe to be true, even in many an office not filled by "white, straight men".