I enjoyed this, but the title and concept are a bit misleading.
I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and did continual comparisons of the success rate to see how different changes affected the result.
All this guy did was take a homeless guy's sign and add a really cool marketing idea to it, which increased the homeless guy's take. Yes, the results of his changes were awesome, but it really had nothing to do with "A/B testing" per se.
I can't find it now, but I read a great article about different signs that panhandlers "fly". One guy flipped his sign depending on the age of the closest likely target. The old person side asked for help and had "God Bless You". The young person side had "I just need a beer".
But the best I've seen was "I bet you can't hit me with a quarter".
Would A/B testing encompass changes as large as what happened here? Having never done A/B testing, it sounds like an iterative process. Would continuous A/B testing ever converge on a solution like the one arrived here?
I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).
Yes, but not encouraged. The literature, or most common applications of A/B testing, is about little improvements. Phrases being changed, picture alignment, that kinda of thing.
Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal.
So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand?
That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.
I agree that the blog post could have contained more information, perhaps some pretty graphs on the conversions per hour with sign 1 vs sign 2. Maybe taking 2 different signs, one with a red cup, one with a blue cup and A/B testing.
But the underlying thesis is still valid. Testing multiple scenarios returns positive changes, and further showing an interesting application that I have never thought about -- helping the homeless with it.
If you're truly data-driven, the yes there is an absolute need for A/B testing. The thing about behavioral psychology is that behavior can be quite different given different variables. I'm constantly surprised when I test things that I'm absolutely sure will work... that don't.
I am not sure if this is what you feel or not (and this may be over-analyzing), but for many people, homelessness is a problem that should be delegated for others to do something about. They don't like the presence of them in their neighborhoods (for whatever reasons) and they often have misconceptions of homelessness that exacerbate the problem.
What works for me is to just chat with them. They don't all beg, but even if they do, one thing they all have in common is the need for ordinary conversation.
The fact that someone posted a story about A/B testing a homeless person's sign sends me two messages. a) They are thinking about homelessness and want to do something about it. b) They are trying to relate it to another problem that many other (non-homeless) people experience. I'm not sure what the author's motivations are, but they definitely got their message across, and they will definitely get people talking about it.
i don't think that it's that, although i am aware it's a (subconscious) possibility.
rather, it seems like it's connected to relative levels of power / wealth. i am wary of doing anything that "uses" someone when there's a large difference (in power / wealth) because it would be so easy to coerce them.
one way in which i can draw a difference: i really don't think this involved "ordinary conversation" (and if i am wrong, and it was - obviously there is no way for me to know for certain - then that would make me much more comfortable; it's also true that i am a poor communicator and i may be projecting my lack of ability...).
"Before you get all 'ugh, you are mocking a homeless man' on me, realize this... This experiment improved this bums ROI"
I got the same feeling. It's because he doesn't treat the guy like a person; he never calls the homeless man by his name, Keith, but keeps referring to him as a bum.
I feel like he did more harm than good. Panhandling, unless extraordinarily effective and coupled with a lack of substance abuse problems (all too common with the homeless) will not pull this man out of poverty for good, or even in the medium-term.
Giving this man more money, but without coupling that with professional assistance, may do him a lot more harm than good.
There's a reason why real academic studies have to pass through an ethics review, and I take issue with how cavalier this guy is about his study.
If the author of the blog has experience working with the homeless, then ignore all I've said above. But otherwise, he's operating way outside his field of expertise involving people and situations he doesn't understand, and complexities he does not seem to be aware of. This is dangerous.
And you can't seem to make your point without name-calling. Kudos.
This is where even a cursory familiarity with the issues surrounding homelessness in your area would be greatly helpful. A huge portion of the homeless population has severe substance abuse problems and/or mental health issues. I highly doubt a bum is going to be spending his panhandling dollars on hookers, but alcohol and blow is more than likely. Given what I know of the proportions through volunteer work, it is entirely unsafe to by default assume a homeless person does not have significant substance abuse problems. Sadly, this is one part of the homeless stereotype that is rooted in reality.
This isn't to pass judgment, but rather to point out that if you really want to help people get off the streets for good, and bootstrap themselves out of poverty, you need to do a lot more than just give money. For a great many homeless people the cause of their homelessness is a lot more than just being down on their luck, it has to do with long-standing problems - substance abuse, toxic environment, and other negative influences. Giving money without resolving these other external issues will just exacerbate the problem.
And note that we're not talking about poor people, we're talking about the homeless. These are people who can't show up to an interview showered with a clean t-shirt - they are not at all like the "merely" poor. Do these guys make enough from panhandling to afford even a shitty room somewhere? Odds are, no, and that's pretty much a bare minimum requirement if your goal is to get off the streets, clean up, and get employed. I contribute to organizations that do this for this very reason - my money is doing much more good (heck, it's at least not doing bad) in the hands of professionals who are able to offer an environment where people have access to the help they need, and the stability to make lasting changes in their life.
And FYI, I'm contributing significant amounts of my money and time to this particular issue, a lot more than change most people give to the homeless in any given year. What are you doing about it?
Strong agree. Everyone seems to excuse their lack of charity by assuming the person who needs help will waste the money on hookers and blow. Maybe there is a significant contingent that does, statistically, but there's no way to know if the particular bum right in front of you right now is going to do that or not.
Is it really so preposterous to assume that this guy's medical expenses caused his financial difficulties instead of assuming that he wasted everything he had on drugs? Sounds believable to me.
leaving aside the issues of 'privilege' for a moment, I think you might have something of a point; I'm watching one of my friends start a startup that will almost certainly crash and burn because he's writing all his business plans to say what he thinks the investors want to hear, rather than coming up with a workable plan.
It'd be quite difficult to answer this question in a scientific manner, but I bet that large investments kill more businesses than they help (vs. bootstrapping)
I didn't get that feeling at all. It seemed like he referred to him as the "homeless man". Either way, I felt he focused more on the problem of A/B testing than making any denigrating comments about the homeless.
if he was "making [...] denigrating comments about the homeless" then it would be obviously wrong. that's really not what we are discussing here - it's certainly not what i felt (i would have simply flamed away if that were the case).
in general, saying "it's not [some extreme caricature of a position]" isn't much use in arguing against anything, and doesn't really help anyone much...
Overlaying the company logo on a different place on the homeless dude's coat in every picture is creepy. That's a person, not a billboard or a wall in the subway.
Um, no. No matter what position Keith takes, the logo is always perfectly horizontal. And in the last image, the logo is more than twice as large as in the other photos. Definitely obviously 'shopped.
Yes, I know what a watermark is. There is a watermark on the photos and you have to look closely to see it. The logo on the coat is not a watermark, since watermarks are not opaque.
I had the same feeling. I think it's because, as a reader, I am unconvinced that the guy is deserving of more money. He might be, or he might be one of the many homeless that have basically chosen to be homeless. My default is to suspect the latter, so his statement comes across to me as better enabling this guy to milk money out of people. Mind you, he might truly be in need, but the article does not demonstrate that. It merely assumes that giving money to the homeless is good, which I disagree with.
Note: Helping those who wish to not be homeless I support; but I don't think that simply giving them money will achieve that.
I was disappointed not to be informed of the experimental method. Did he normalize the data for each time period? (Dollars/Person within 20 feet), or is it unnormalized and we could just be looking at a situation where more people came later. How long were the time periods of each trial? How much better did he do at each trial, in terms of percentages?
Also, since he apparently has the data, I'm dieing to know how much one of these guys does make per day.
"The first thing we always do is make slight modifications to what we already have."
rather contrasts with
"The next big difference is that we changed colors and went from cardboard to white to spark the interest of people walking by instead of automatically having negative associations that they have with cardboard and homeless people."
The second can (and in the context of the article maybe should) be tested. The small change is to make exactly the same text on the different material.
Forget the whole a/b testing thing. The real value of this blog post is that it shows a great way to utilize personal skills to help someone else. That's the real genius, out of the box thinking here.
I often get too wrapped up in the idea of A/B testing as a digital concept. After reading I'm starting to think of all the "everyday" items that can be optimized with a little testing.
That said, I am now researching a way to add hand sanitizer to my website.
One reason to do A/B testing is to collect data/facts.
That article contains only opinions ie : "The biggest difference, is that we are now introducing a bribe"
Trust data, not opinions.
I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and did continual comparisons of the success rate to see how different changes affected the result.
All this guy did was take a homeless guy's sign and add a really cool marketing idea to it, which increased the homeless guy's take. Yes, the results of his changes were awesome, but it really had nothing to do with "A/B testing" per se.