"You don't look like us" - Illegal.

"We don't think you fit in with our culture" - Legal.

I think this is why any crime where the intention is part of it is so hard to prove, like discrimination, hate crimes, etc.