I think you're reading much too far into this situation. At no point did he say anything resembling "refusing to engage with coworkers of a different gender".