Definitely like the second part. It would seem that if you aren't actually using your patent, the patent's purpose is lost.
I also think there should be oversight on the patents that are granted. Right now you can patent something that has 20 years of prior art with relative ease. That's silly.
There _is_ the doctrine of laches, a defense in the case of suits brought after a long period of non-enforcement. As with most defenses, this entails lawyering up and going to court, part of what makes this process so expensive.
There were also reforms in the 1990s which extended the life of a patent (from 17 years to 20), but started the clock at filing time, not grant. This was due to a slew of patents issued to Jerome Lemelson which he kept appealing to the patent office. These were eventually granted, a practice termed "submarine patents". Lemelson (and his heirs) were awarded over $1.3 billion in royalties (he's also got a wing of the Smithsonian named after him).
I also think there should be oversight on the patents that are granted. Right now you can patent something that has 20 years of prior art with relative ease. That's silly.