> I would bet it is some ML model to detect fraud. This model will never be perfect – it will have false positives. My guess is – while picking the precision/recall thresholds for blocking users, someone higher up would have argued it is okay to cause some false positives to prevent a lot of harm.
> And they would have justified to themselves saying there's always recourse through customer support. But customer support tools to investigate and un-ban users would be slow and painful and lacks capabilities needed to check if the complaining user even passes the basic smell tests for a fraudster. And nobody can really explain why the model blocked the user in the first place. There's no well-lit path from CS to engineering on a case-by-case basis. Escalation would happen in bulk/batches – when lots of seemingly 'innocent' users complain to CS, CS may escalate to engineering.
Can such a model be trained effectively if isolated reports of false positives are rejected without meaningful investigation? In that case, wouldn't the model be trained with bad data?
What if for each reported false positive there are more users affected who didn't report it (because their accounts were less valuable, because their time was more valuable, because they were too upset, because they were too timid, because they died (for unrelated reasons), etc)?
> Can such a model be trained effectively if isolated reports of false positives are rejected without meaningful investigation?
No, and much like YouTube auto terminating accounts with 10 years of content there is absolutely no excuse for certain very obvious cases to be handled without human interaction. There absolutely must be flags that stop terminations without a human.
> And they would have justified to themselves saying there's always recourse through customer support. But customer support tools to investigate and un-ban users would be slow and painful and lacks capabilities needed to check if the complaining user even passes the basic smell tests for a fraudster. And nobody can really explain why the model blocked the user in the first place. There's no well-lit path from CS to engineering on a case-by-case basis. Escalation would happen in bulk/batches – when lots of seemingly 'innocent' users complain to CS, CS may escalate to engineering.
Can such a model be trained effectively if isolated reports of false positives are rejected without meaningful investigation? In that case, wouldn't the model be trained with bad data?
What if for each reported false positive there are more users affected who didn't report it (because their accounts were less valuable, because their time was more valuable, because they were too upset, because they were too timid, because they died (for unrelated reasons), etc)?