A mathematics PhD is going to be in the far right tail of aptitude for a lot of relevant skills. This is the admittedly very rare case where altogether tiny differences in "inherent" ability will get amplified to the point where they actually start to matter - whatever "inherent" might mean, in the first place! Yup, crucially, even differences that are generally understood as "not inherent" can be so hard to address that they might as well be, for all we can do about them. If you ate too many lead paint flakes as a kid, the probability that you're going to be a successful math PhD drops quite a bit. And, short of tearing down and rebuilding a lot of homes that happen to have lead paint, lead pipes, etc. in them, we're unfortunately not going to be able to address this "inherent" unbalance for the foreseeable future.

You understand that the perception that you're looking at someone's skin color or facial features, making a prediction as to whether they ingested lead paint as a child, and then acting accordingly is going to be problematic, right?

But of course no one sane is doing anything of the sort. The issue is how to explain the difference between "% of people with visible feature X in the general population" and "% of folks with feature X who happen to be math PhD's". Well, if feature X correlates with issues like eating lead paint as a kid, even to a small, otherwise undetectable extent, you're going to find that sort of difference.