One way to look at it is minority groups who can’t turn off being visible minorities can be different than people who may be minorities in a a few areas of life but still have access to benefit from privilege in others.

I dont believe its good to narrow down minority status into something which is only visible, feels a bit ableist.