The answer to that varies from person to person, and it has varied in aggregate over time, but the effect on those who remain has been the same. It doesn't matter whether those who left were/are fleeing actual danger, race-based perception of danger, or pure economic rationality. If you tip a marble on to a sloped ramp, it will continue rolling down even without further impulse. The fact that at one time race and racism were significant drivers of this effect is sufficient to justify their consideration in any present discussion of causes and/or remedies.