You are right that context is everything, but it works both ways.

You can make a sexist joke without being sexist, but it can still be inapropriate/unprofessional, and it can still cost you dearly.