That is precisely what is wrong with the assumption. Diversity is not just skin color as Americans think. I have people of color as neighbors who I share more cultural traits with that neighbors who are white (I am white) My wife is of Asian decent but I have more in common with her than with many other white Western Europeans.

Diversity is also language, culture, language, identity etc.