I agree. The penalty on the user of AI would in turn incentivize the user to penalize/reward the AI developer. But I’m not sure if it’s reasonable to require every user to be able to assess the long run accuracy of a statistical system given their limited interaction. Maybe there’s a market for companies to reliably assess and certify AI systems or by creating a sort of a prediction market that penalize overconfident false positives.