Good journals already have some of these features, private or community driven. E.g. "Quantum Journal" is community driven, and while it lacks polish, it has a fairly functional editorial board. APS journals, which are much more polished (but also require submission and/or subscription fees), have pretty serious internal reviewer database (including editors evaluating the quality of the reviews).