Hacker News new | past | comments | ask | show | jobs | submit login

Interesting. I'd like to know more about how you setup your experiments.



Have chucked the simulation code up on github: https://github.com/richardfergie/split-test-simulator

Current things that I think are wrong with it:

* Successive tests should not be independent; picking the wrong option should have repercussions beyond the next test

* The distribution of conversion rates from which samples are drawn was picked because it is convenient not because it is correct

* Takes no account of resource issues around creating new variations e.g. it is unlikely that people can create a new variation every day but some strategies assume they can.

Please let me know your thoughts




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: