Check out our interview on Machine Learning Street Talk with Jonathan Frankle explaining Rewinding and why you can only reset the Learning Rate rather than the weights!
https://www.youtube.com/watch?v=SfjJoevBbjU&t=1177s
Interesting, he thinks "there has not been much progress in pruning NNs in the last 10 years". Maybe he meant that it's often possible to train a non-pruned network of the same size as the pruned network, and achieve a similar accuracy.
