> Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the people within the group who are better or worse risks.

No model is perfect. Odds are you can better predict risk by (directly or indirectly) attaching bonuses or penalties to given races, all else being equal.

e.g. Say, irrespective of all other measured quantities, people of race A are more likely to default because of other systemic racism against them. As a lender, it'd be completely rational to consider this and make "better" choices. And if you're not allowed to measure race, it'd be completely rational to find other variables that don't have an obvious causal relationship to credit risk, but predict race and thus have some information about credit risk. This is the exact kind of thing ML does.

Then, in turn, this becomes self-reinforcing. Because other institutions discriminate against race A, the risks going forward of dealing with race A increase...

One possible approach is to make a model that besides its intended prediction also predicts race, and penalise its ability to predict race with probability larger than random. If the model has representations that can't predict race, then it will have to use non-biased features to make its predictions. But I guess in many situations this would degrade the accuracy and it will not be welcome from a business point of view.

How would one do this? If you have a ML model with outputs classifying credit risk, and outputs classifying race, it's easy to learn coefficients that are bad at classifying race but still take race into account in classifying credit risk.

That is, what's your loss function and how does it prevent race from being considered in the credit decision?

It's a setup that is somewhat similar to GANs (and even closer to a related method called Fader Networks):

- a first network take the input data and return a representation A (like an embedding vector): let's called it the "censor network"

- a second network take this embedding A as input and is trained to predict the class that should be censored (for example the gender of a person) : the "discriminator network"

- a third network take the same embedding A as input and is trained to predict the real task of interest (for example the probability of credit default) : the "predictor network"

The idea is that, by training the censor to make the discriminator fail (predict the wrong class) while making the predictor work, it will force the censor to learn a transformation of the input data that keeps the task related information in the embedding A, but removes the information correlated to the "censored class" (and that could be used to discriminate).

Here's a reference about this kind of methods, but it's still an active domain of study in ML and there are many papers that followed this one: https://arxiv.org/pdf/1801.07593.pdf

Neat. But might it not just be easier to predict the influence of race and then use that to adjust the output/threshold?

>No model is perfect

But some are definitely better than others.

It sounds like you're looking at things purely from the point of view of getting the correct average for a group. But whether or not you get the correct average doesn't tell you if you're using a good enough or the best available model completely apart from fairness or justice.

If you do some type of testing and you know 5% of the tests should come back positive, is there a difference between reporting 5% at random and actually doing the tests? Of course!

> It sounds like you're looking at things purely from the point of view of getting the correct average for a group

No, I'm looking at things from the point of view of making a model that fits well to the original dataset and then is verified in the actual accuracy it makes over time.

If adding race-- or inferring race-- makes the model substantially better in predicting outcomes, is it right to do so? Credit default risk is correlated to race, even controlling for other variables. Hence, using race would help you make more accurate predictions.

When you say "better", better than what? How can your model tell you that some other data would not work better? The fact that you are looking at a correlation tells you that it's one of many ways to infer what you want to determine, and that it's imperfect. This is logically certain if we agree that race is not causal. So the only question is how much better is another model with different inputs.

I'd argue that curve fitting isn't modeling. And when inserting a fitted curve into a feedback system you're very likely to just perpetuate the problem you're looking to eliminate.

> I'd argue that curve fitting isn't modeling.

This entire discussion is about ML models, which can often be fairly described as very fancy curve fitting.

> And when inserting a fitted curve into a feedback system you're very likely to just perpetuate the problem you're looking to eliminate.

This is not a problem for the individual credit issuer-- they're not looking to eliminate the problem. They've avoided some credit risk by taking race into account. They've improved their expected value, even if society is stuck with the cost of the problem getting worse.

For credit reporting this sort of thing creates bad enough externalizes that it needs to be outlawed.

And I repeat curve fitting isn't modeling. Because in that case it's not a model it's a prescribed outcome.

But what is your definition of perfect (or better). That it perfectly aligns with societies moral view, or that it makes the company the most money.