I like the premise of this a lot, but it seems to me that the setting that the author chose (some UI element of a website) is one of the worst possible settings for this: what matters a whole lot more than if your button is red or green or blue is some modicum of consistency.
If you're constantly changing the button color, size, location, whatever... that is an awful experience in and of itself, is it not? If the Amazon "buy now" button changed size / shape / position / color every time I went to buy something, I would get frustrated with it pretty quickly.
One aspect of testing they leave unsaid is you identify your users (cookie, most commonly) to make sure each user always gets the same experience. That's why your numbers are all based on unique users, not merely users.
Their experience will still change once their cookie expires, but that amount of time is completely under your control.
If you cookie the users and make sure the experience they saw is persistent that solves most of this problem. But if you run a lot of separate tests than it's hard to avoid this.
If you're constantly changing the button color, size, location, whatever... that is an awful experience in and of itself, is it not? If the Amazon "buy now" button changed size / shape / position / color every time I went to buy something, I would get frustrated with it pretty quickly.