There are also a lot of people who would really like to improve upon existing products but are prevented from doing so due to patents. So patents promote some kinds of innovation while severely restricting other kinds. It may well be true that there is more potential innovation being blocked due to IP restrictions than there is being promoted by IP rights. People seem to take it as a given that the latter outweighs the former, but rarely have I encountered evidence for this claim. The implications of getting this wrong are staggering, and I worry that people rarely critique this idea.
Maybe make it so that the patent royalties are paid by the consumer as patent tax. Then that tax is distributed among all patents in use by that product.