> your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science
But all science works like this. All scientific models can have errors, and presumably they all do. Even really basic empirical science is only reliable to the extent that our models of how photons and our visual systems work, and those models have errors we know about and probably others we don’t know about.
Fallibility does not mean that something is not science, on the contrary, denying that some theory or model is fallible is profoundly unscientific.
But of course, that doesn’t mean we should accept “black box” algorithms as the end of the story. We should strive to develop explanations for those things just like for all other things.
But all science works like this. All scientific models can have errors, and presumably they all do. Even really basic empirical science is only reliable to the extent that our models of how photons and our visual systems work, and those models have errors we know about and probably others we don’t know about.
Fallibility does not mean that something is not science, on the contrary, denying that some theory or model is fallible is profoundly unscientific.
But of course, that doesn’t mean we should accept “black box” algorithms as the end of the story. We should strive to develop explanations for those things just like for all other things.