You don't need everyone to be leaving it, you just need the new generations of scientists/entrepreneurs/etc to go somewhere else.

And if they can't get in the US, well that's an easy call.