While the notion of treating these systems as “sociotechical” rather than purely technical is probably a good move wrt actually improving people’s lives, I can say from my own experience in academia that there are still way too many academics working in this field who don’t think it’s their problem. I’ve personally raised these types of issues before and been told “we’re computer scientists, not social scientists”, as if “social scientist” is a derogatory term. The biggest impediment here is, in my opinion, overcoming the bloated egos of the people who think the social impacts of their work are somehow out of scope. All is well as long as you can continue to publish.
There are way too many people that conflate MSE or other abstract technical measurements of model performance like they actually represent the impact any model has on a problem. Even if we could somehow perfectly predict an actual realization instead of a conditional expectation that still forgets to ask the question of why we predicted that. Are we exploiting systemic biases, like historically racist policies? Almost definitely (unless we've consciously tried to adjust for them, and still we've probably done that incorrectly). I've become much less interested in models that basically just interpolate (very well I might add), and more in frameworks that attempt to answer why we see particular patterns.
