This isn't a hypothetical discussion. Asymmetric encryption has been battle-tested for decades now. If it's as weak as you say, SSH and GPG would have gotten blown open long ago rather than being the the thing we all reach for when we wanted an actually unbreakable system.