This is something that I had been thinking would greatly benefit the arXiv for a long time.
Presently, when you do a scientific work, the article goes to a referee who then sends you back their comments which you account for before resubmitting. Sometimes you get an excellent referee who really knows his stuff and gives reasonable comments for improvements. Sometimes you get a guy who really just can't be bothered who gives minimal comments leading you to wonder if they've even read it. Sometimes you get an opposing group, which frequently leads to untenable comments and prompts submission to a different journal.
This is the only feedback you will ever get outside of your coauthors except the citation count. In my opinion it would be amazing to have some big named authors who have read your paper drop off advice, what they liked what they didn't like, etc. At most universities in most groups you do "journal club" once a week where you discuss others' papers and produce this exact feedback, but there's no forum to post it in, so it just stays in the journal club.
However, just as abuse on arXiv led to the transformation to an invitation only site (I forget if you need an invite or just a university sponsored email; see also vixra.com), the community on a site like this _should require your real identity_.
It could be devastating to a young researcher to have their work publicly shamed by an anonymous commenter who has it out for their research group. But if the comments are linked to real identities, I think the community will police itself... Although there are frequently unofficial "response to... " articles on the arXiv, they are publicly attached to other research groups, and you will sometimes see "response to response to ..." letters.
It's interesting to see these sort of 2010 things popping up amongst our 1990s bastion websites like arxiv and ADS and such (see researchgate, the facebook of scientists). But frequently they kind of seem to fall victim to the same downfalls of their non-scientific counterparts ("cite" is the equivalent of "like" on researchgate to improve your "profile impact" metric so you frequently get people acting needy about "citation requests" even though we have our own metrics like Hirsch Indeces to measure scientific productiveness in an objective way).
TLDR;
I worry that a comment based website could host troll-like behavior which could be especially harmful when the whole premise is people's professional work. This is one of the few places on the internet where I think real names must be required and institutional affiliation should be provided (as is the case with arXiv). As it is I signed up with a BS name and email in 5 seconds and can immediately start trashing this paper on quantum physics that I know nothing about.
Actually, as a young researcher, I find the complete opposite. Sites like researchgate or facebook or any other "comment-ie" thing is sort of superficial in the sense that even though you might get some comments, they're usually quite minimal and bad. On the other hand, reviewers for conferences/journals are sort of obliged to put time into it and dedicate to give you a proper review and most reviews I've received are of very high quality.
That is true, but ref reports can definitely be nightmares, especially in niche areas where only 3-4 groups qualify to ref something. And some journals can be particularly bad about the feedback, especially some of the lower tier journals, but I guess that's why some people go to the low impact journals...
I would also guess that most social media like comments would be "have you seen my paper vaguely related the topic", which is not ultra-useful.
But if you could somehow emulate the conference feel, where people are constantly walking up to you and saying "oh this is neat, I think Prof. X is working on something similar, you should find him," that would be optimal.
The comments SciRate gets at the moment are pretty decent, I think because people mostly just integrate it into their existing review process. https://scirate.com/arxiv/1501.07071 is a good example. If the community were to grow though we'd definitely want heavy moderation.
My general feeling after many years of internet use was absolutely to limit anonymity (I'm the current SciRate maintainer). Scientific discourse has its own culture, though, which is a bit different to internet culture in general. And the users seem to like anonymous peer review; we've had requests to make it more anonymous: https://github.com/scirate/scirate/issues/304
So currently I'm adopting a "wait and see" approach wrt. trolling. Stay open unless/until it becomes an issue, but make sure to crack down swiftly if it does. For the moment, we do have comment reports/downvotes and moderators.
You just need a mechanism to weed out troll behavior. Maybe a few moderator users. As researchgate shows you don't want to use real names; people will not give honest feedback. Dont forget science has a lot of politics. Besides, anonymity works well with journal peer reviews, why change that?
I think it's fairly easy to give honest feedback about a paper you dislike without it being politically disastrous, just something along the lines of: "Is there a reason why you used method a instead of method b?"
It hasn't been long enough to see how these specific groups would act with anonymity, and certainly some groups are more resistant to trolling than others, but I think in general anonymity will tend toward the nasty part of the spectrum.
Even in public, recorded lectures, I've seen full professors just go to town on the speaker trashing their work, especially if the work is controversial in any way. It's nice to be able to see who's acting that way.
A friend of mine had a paper response written to his first ever paper that point-by-point tried to discredit his whole point. Then his professor had to step in and write a point by point rebuttal, because someone with just one paper has no clout to deal with that really. It's nice to see that happen in public because now the guy just looks like an ass for jumping on first year PhD students.
There is also PubPeer, which allows anonymous commenting on articles, including arXiv preprints. It has been fairly successful -- its commenters have uncovered several cases of fraud and appear to be fairly active:
So far it seems that scirate is used just for the "like" button, which is the one big missing feature of the arxiv. (And probably should not be implemented there anyway.) Most science trolling still seems to happen in a very public & attributed way, via those articles with title "A note on..."
I'm not sure how it is in other fields, but in Astronomy we have a monolithic database of publications (called ADS) where you can search by author or title or date or keywords etc. And there are sort options for views, downloads, citations, etc (all with time data, so you can see trending behaviors). IMO that serves the same function as "Like" but better.
E.g. http://arxiv.org/pdf/0909.4006.pdf section 1.1 should have floor function instead of brackets IMO.