If you indefinitely grow the collection of packages that you have in use in permanent virtual environments, then yes your disk usage will and should grow indefinitely anyway.
There's not much point recycling contents from a package cache unless you currently don't have a venv using that package and also don't reasonably expect to have one in the near future.
And I'm saying this as the guy complaining all the time about things like the size of Numpy wheels.
