> Seems straightforward to compare open-source vs closed-source bug fix performance.
You can publish a best-paper in ICSE if you could pull this off. There are so many things that make this challenging. For starters, we don't even have the ground truth for what bugs exist. Even just looking at bugs we've already skewed our process dramatically based on the various different development processes of different projects.
You are right to question how many eyes are on typical random libraries. The answer is zero. Even huge libraries have extremely few eyes on them. When it comes to "many eyes" it is actually basically just the linux kernel and a very small number of other projects that get this sort of attention. The large majority of all open source projects, even those used by millions of projects, get zero meaningful attention beyond "hey I threw my tool at everything on github and spammed owners with nearly useless reports."
You can publish a best-paper in ICSE if you could pull this off. There are so many things that make this challenging. For starters, we don't even have the ground truth for what bugs exist. Even just looking at bugs we've already skewed our process dramatically based on the various different development processes of different projects.
You are right to question how many eyes are on typical random libraries. The answer is zero. Even huge libraries have extremely few eyes on them. When it comes to "many eyes" it is actually basically just the linux kernel and a very small number of other projects that get this sort of attention. The large majority of all open source projects, even those used by millions of projects, get zero meaningful attention beyond "hey I threw my tool at everything on github and spammed owners with nearly useless reports."