While interesting, I think that this is more of a mathematical proof for something that people doing any sort of testing should remember:
Don't stop before the test is complete, just because you've gotten an answer.
I generally leave my A/B tests up well after I've gotten a significance report, mostly because I'm lazy but also because I know that given enough time and enough entries, the significance reports can change.
Especially in the multi-variate tests that Evan wrote about, just because you get one result as significant doesn't preclude other possibilities from also being significant.
Don't stop before the test is complete, just because you've gotten an answer.
I generally leave my A/B tests up well after I've gotten a significance report, mostly because I'm lazy but also because I know that given enough time and enough entries, the significance reports can change.
Especially in the multi-variate tests that Evan wrote about, just because you get one result as significant doesn't preclude other possibilities from also being significant.