>> the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %
> The chance of each winner being genuine is still 95%
Not really. It depends on what’s the unknown (but fixed in a frequentist analysis like this one) difference between the options - or absence thereof.
If there is no real difference it’s 100% noise and each winner is genuine with probability 0%. If the difference is huge the first number is close to 0% and the second number is close to 100%.
You're right, I made the classic frequentist statistics mistake. All we can talk about with certainty are the probabilities of things happening under a specific null model.
> The chance of each winner being genuine is still 95%
Not really. It depends on what’s the unknown (but fixed in a frequentist analysis like this one) difference between the options - or absence thereof.
If there is no real difference it’s 100% noise and each winner is genuine with probability 0%. If the difference is huge the first number is close to 0% and the second number is close to 100%.