I think that an issue with that method is the barrier to entry. To make a change to an OSS project, you need as little as a text editor. Code something up, try a few things, commit (and add your name to the list!).
To make a change to someone else's article, all you can do is question their methods, and question them again more sternly. It's not trivial to repeat experiments with scientific rigor, but it is (generally) trivial to test out a code change or two.
Another issue is experience - it takes a lot of education to get someone to the point of being able to apply scientific rigor; there's a lot of seductive pitfalls along the way. It's not really something you can be 'amateur' about. That's not really the case with a lot of OSS, where you can successfully contribute without understanding important but subtle issues.
Well, I certainly don't agree with your description of what software development entails. You need a text editor, but you also need to be familiar with a whole ecosystem of tools, libraries, platforms, procedures, and general technical knowhow. You need to commit your changes; you also need the project owner to think what you've done is worth merging into the main branch. The suggestion that it's something you can "be amateur about" is making me giggle.
If anything, it's the development behind closed doors that you can "be amateur about". Work done in public is too embarrassing and too unlikely to attract any interest to do a bad job on.
But that's kind of beside the point. Even if the expertise is rarer on research, I don't understand why that matters. Wouldn't that be an argument for pooling the resources you do have? I mean, if there are only four people working in a field, wouldn't it be better if they all could critique a new work, and if they were able to watch it progress and influence its direction (maybe stopping mistakes early!), and if the judgement of a new work came with their written opinions on its worthiness attached?
turns out my original response was eaten by the daft HN 'unknown or expired link' thing.
but in short: It seems you misunderstand me. They both require understanding, but it is easier to effect changes on an OSS project than it is to make an author go back and rewrite or retest their article.
Then you're simply increasing granularity...instead of judging papers by what journals considered them worthy, judge by what individuals considered them worthy...and if you don't know the individuals involved, the reputation system is there to help you.
It'd have to be something similar to pagerank, not a simple vote-count method. High-reputation individuals should be able to contribute more to the reputation of other individuals.
Another idea would be to apply pagerank to article citations, though this would only be useful for older papers.
Pagerank patent expires in 2018. There are some other interesting reputation system out there, too.
For programming reputation has a point, but for science articles, each should be judged on their own merit. Ideally you shouldn't know whose article it is that you're reviewing, or you run the risk of preferential treatment.
I wasn't thinking of judging articles by the reputation of the authors. Instead, I was thinking of judging the value of reviews by the reputation of the reviewers.
I think that's actually pretty similar to what we're doing now. Articles accepted by journals of high reputation get a high reputation, and the journals select high-reputation individuals to do reviews. A good distributed reputation algorithm could perform essentially the same function. Leaving review open to anyone could help preserve net objectivity even if some individual reviewers are biased.
But anonymity could be accomplished by releasing articles initially with a timestamp and a digital signature by a new public key. After review, the authors could reveal that they have the matching private key.
To make a change to someone else's article, all you can do is question their methods, and question them again more sternly. It's not trivial to repeat experiments with scientific rigor, but it is (generally) trivial to test out a code change or two.
Another issue is experience - it takes a lot of education to get someone to the point of being able to apply scientific rigor; there's a lot of seductive pitfalls along the way. It's not really something you can be 'amateur' about. That's not really the case with a lot of OSS, where you can successfully contribute without understanding important but subtle issues.