I feel that there is the danger here of a blind leading the blind effect. I tried it out for giggles. Not to pick on my reviewer, but here:
There are alot of words on the landing page [of BCC]. Look at some of the ycombinator pages and you will notice the average site has about 45 words and 4-9 links. Don't give them so many options and so much to read. Otherewise, looks good!
The quality of your users' interactions with FeedbackRoulette is directly related to the quality of the advice they receive. To recruit and retain the best users who give the best advice, you have to be giving out advice which is worth their time. The above quoted advice is well-intentioned but does not convince me that I am likely to realize business value by continuing to be engaged with the service.
To see another take at resolving this problem, take a look at StackOverflow: by breaking the 1:1 nature of the communication, you can use people's desire to peacock to get hypercontributing members of the community to be responsible for most of the good advice on the site, and let the 90% who do no writing merely vote the good stuff to the top. Hopefully they pick the right good stuff. If, on the other hand, StackOverflow picked two random programmers with a question and connected them, the average experience of using it would be getting your query on Rails ActiveRecord validation answered by a junior Java programmer from India. That would be a much less effective StackOverflow. It would likely not maintain the interest of somebody like John Skeet (who, if you are not familiar with the SO community, is something of a local legend).
(Sidenote: if this reviewer was right and there are in fact YC companies with 45 words on the front page... how do I put this gently... there must be someone in your alumni network who you trust on SEO, right? Talk to him about whether that is a good idea. It will be a very brief conversation.)
It occurs to me that you could literally retrospectively measure the odds that someone's first experience with the site was due to a hypercontributing outlier member. My intuition is that a) they contribute a hugely disproportionate amount of the value on SO (pick any metric you want for this), b) they contribute a hugely disproportionate amount of value relevant to someone's first use experience, and c) if you segment users into "first use was influenced by a hypercontributor" and "first use was not influenced by a hypercontributor", there is a measurable difference in retention rates in favor of the first group.
Proving or disproving these assumptions would be pretty important for strategic direction at a QA site, incidentally, since the world if I'm right looks very different from the world if I'm wrong. Since StackOverflow's data set is public, you could do it as a weekend project. In general, though, this is the kind of stupidly powerful thing you can go with the data you already have in your startup.
I think your intuition is correct - that user contributions generally follow a power law, so when you randomize 1-to-1 interaction among all members, the likelihood of a person in the long-tail getting feedback from another user in the long-tail is high. A good essay on this: http://www.shirky.com/writings/powerlaw_weblog.html
Indeed, this seems like the wrong matching model if you want either (a) expert review; or (b) representative user review. A random other website-developer isn't especially likely to be either. There is great value to the ad-hoc, user-driven matching of people with topics that occurs on sites with a public new-queue and topic-leaderboard.
And so this reminds me of an idea for a more focused version of HN I'd had a while back. Let's call it 'Bluntr' for now. All stories are requests for review of work: code, designs, websites, products, whatever. All comments are supposed to be brutally direct feedback, with as much concrete backup-data/recommendations as possible.
Note that this does not mean 'nasty': no mockery or vague insulting filler words would be permitted. But it also means every thread can jump straight to the point, without traditional 'attaboy!' or 'I like the X but...' niceties slowing things down. Essentially: at Bluntr, empty compliments or insults would both be flagged/downvoted to oblivion, while steely-eyed evaluation would be promoted.
The community might not gel without those normal verbal ticks and grooming-activities, but then again these ground rules might help the really focused, strong contributors cut to the chase a lot faster. You are not allowed to be either nice or nasty, just honest -- and so you are free to be more productive. That's the initial theory to be tested, at least.
> the average experience of using it would be getting your query on Rails ActiveRecord validation answered by a junior Java programmer from India
Please avoid such racist stereotyping. There are terrible and awesome programmers in every community, and it's an insult to put them in the same bin and call them the same name. I found it quite offensive myself.
That reminds me of what my cabbie said when I asked if there was much crime around here (this was in Jamaica). He responded, "there is crime in every country." He was clearly offended, so I decided not to press on with discussions of statistics.
I like the idea. I think there should be a skip button as there are a couple sites that I wasn't able to review properly (one specifically asked for people to review it in another language and the other required me to sign in).
I also think it'd be cool if the homepage showcased some of the highest rated sites, like the way Logopond (http://logopond.com/) works for logos.
Agree with this. I cancelled a review because I simply can't give very good feedback on the type of site I was given. I waited a while, clicked 'Review Next' again and got the same site.
If the creator of FeedbackRoulette is reading, skip button pronto!
I reviewed 4 sites just for kicks and submitted my site to it. So far, only 1 person rated my review so I got 1 feedback point and 1 review for my website. Seems like getting an ROI on my time is dependent on a) whether people see the rating buttons b) click on them. Seems like something that needs to be fixed.
Reviewers submit a short review with the basic findings and points of improvement, and hint at a more in-depth review.
For a nominal fee, you could access a more in-depth review and consulting (maybe with coding help, etc..). The site could be a nexus between people needing reviewing help, and consultants, for a small fee.
Top reviewers could make a living out of this, other people can submit good reviews in the hope of being picked for consulting work.
Just my 2 cents of brainstorming (probably millions of buts to this one).
This really works as a concept. I've signed up and reviewed a couple of sites already. I'm looking at one now that looks suspiciously like spam though. There doesn't appear to be a spam button for when you're looking at a site. If a site isn't spam then being labelled as spam is still useful feedback for them, even if it does mean going back to the drawing board.
Please allow the website owner to rate the review he receives and allow his sites to be reviewed based on the rating of the review that he has given.
So for example: If i rate site A.com and the siteowner of A.com rates my review as 5/5 then my sites should be eligible for 5 reviews. The ratios could be different - essentially reward owners for better reviews.
I see a very clear monetization opportunity here. I pay money and get X number of reviews. I am sure you would have thought about this. Cool stuff.
It would also be nice to be able to talk back with reviewers if they allow for it. I've gotten some interesting comments and I'd like to either ask a few clarifying questions or just get to know them a bit better. A 2-way communication message would be nice.
We submitted our site this morning and have received really good feedback, which we are going to implement on our site. I also reached out to the team with feedback and received a quick thoughtful response. I understand there is a danger to a free product, but people, it's free and you get rapid feedback, you don't have to use the feedback. I definitely think some sort of scoring to get top feed backers up to a higher level (which people could pay for) could create a value proposition.
I wish I could see and possibly review sites without handing over my email address. I'd like to be able to explore the kinds of sites that are posted and possibly the kinds of feedback they are getting before creating an account.
(Also, what if I don't have a website I want reviewed? I guess that violates the 1:1 relationship feedbackroulette is trying to develop, but I'd be interested in what non-"webmaster" folks have to say if there are any willing to submit a review. )
I've posted a couple of decent quality reviews and most have been accepted with 5 star ratings. However, very few reviews are coming back the other way, leaving me feeling like I've somewhat wasted my time.
Perhaps this will change later, but in the meantime, perhaps sites could be circulated around instead of just being assigned to one person (which is what I presume is happening.)
I liked the idea, but I think there should be some sort of options or categorization of websites that you are submitting and you would like to review. I just tried it out and I got a website to review - but it did not have any details on what aspect needs to be reviewed.
You might want to check out the model http://www.critters.org/ uses where members get to pick what they want to critique. I think it might work better than the random-matching approach.
Currently there is an incentive to rate reviews as follows: if you want to increase the probability for that person to review another of your sites, you would give them a rating similar to your rating (which could be quite low).
Interesting concept. What's the business model here? Advertising? Maybe a premium service where you can choose which sites to review and be able to target certain reviewers for your own site.
Raphael, have you added filtering by reviewer's location? Most of responses come from Asia and from people eager to earn your 60c per response. They are positively biased at best, and half-assed and incompetent at worst.
Dear [person whose name I don't know],
You have a right to reject (and automatically request a new response) for any half-assed or incompetent responses.
That said: I receive a lot of positive feedback on the service and in the instances when someone contacts me with a bad experience, I take care of them. I'm a little surprised to see such a pointed comment about FBA on Hacker News. If you used the service and had a poor experience, I wish you would have given me a chance to do something to make it right with you.
We spoke over the email back in July. I emailed you to say that the quality of the feedback had gone substantially down for the reasons I posted above. You replied suggesting to reject as many responses as I felt needed and alternatively to refund the purchase. This is just, and you do in fact take care of the poor experiences.
I rejected good half of responses, but it also made me realize this:
Having thought a bit more about this whole situation here's what I think is the root of the frustration - I would rather not get to the point of needing to reject comments.
There are some obviously half-assed comments, and these are easy to reject. However there are other comments, still useless, that clearly had some time and effort put into them. Rejecting these seems unfair, especially considering the price point. And there is lots of these comments.
That's why an ability to focus the survey more precisely is IMO essential.
Hence - my question. Have you got a chance to add country filters to FA or not?
HN isn't always the right demographic for feedback. I could this particular service improving to limit/filter feedback in a way that would make them more useful.
Also, there are a number of other improvements that would really make this site a lot more valuable for getting feedback than HN, such as rating feedback givers, incorporating Silverback, etc.
There are alot of words on the landing page [of BCC]. Look at some of the ycombinator pages and you will notice the average site has about 45 words and 4-9 links. Don't give them so many options and so much to read. Otherewise, looks good!
The quality of your users' interactions with FeedbackRoulette is directly related to the quality of the advice they receive. To recruit and retain the best users who give the best advice, you have to be giving out advice which is worth their time. The above quoted advice is well-intentioned but does not convince me that I am likely to realize business value by continuing to be engaged with the service.
To see another take at resolving this problem, take a look at StackOverflow: by breaking the 1:1 nature of the communication, you can use people's desire to peacock to get hypercontributing members of the community to be responsible for most of the good advice on the site, and let the 90% who do no writing merely vote the good stuff to the top. Hopefully they pick the right good stuff. If, on the other hand, StackOverflow picked two random programmers with a question and connected them, the average experience of using it would be getting your query on Rails ActiveRecord validation answered by a junior Java programmer from India. That would be a much less effective StackOverflow. It would likely not maintain the interest of somebody like John Skeet (who, if you are not familiar with the SO community, is something of a local legend).
(Sidenote: if this reviewer was right and there are in fact YC companies with 45 words on the front page... how do I put this gently... there must be someone in your alumni network who you trust on SEO, right? Talk to him about whether that is a good idea. It will be a very brief conversation.)