In my experience, there’s just not much depth to the math behind A/B testing. It all comes down to does A or B affect X parameter without negatively affecting Y. This is all basic analysis stuff.
The harsh reality is A/B testing is only an optimization technique. It’s not going to fix fundamental problems with your product or app. In nearly everything I’ve done, it’s been a far better investment to focus on delivering more features and more value. It’s much easier to build a new feature that moves the needle by 1% than it is to polish a turd for 0.5% improvement.
That being said, there are massive exceptions to this. When you’re at scale, fractions of percents can mean multiple millions of dollars of improvements.
The harsh reality is A/B testing is only an optimization technique. It’s not going to fix fundamental problems with your product or app. In nearly everything I’ve done, it’s been a far better investment to focus on delivering more features and more value. It’s much easier to build a new feature that moves the needle by 1% than it is to polish a turd for 0.5% improvement.
That being said, there are massive exceptions to this. When you’re at scale, fractions of percents can mean multiple millions of dollars of improvements.