Well, one way to think about this is: how much abstract algebra are you keeping in your head to reassure yourself of the security of classical asymmetric cryptography? It's surprisingly deep. Some programmers are "comfortable" with it because they've been brought up being taught that "factoring" is just the way asymmetric cryptography works, but that has never really been the whole case.
Elliptic curve is not at all simple to comprehend! It's easy to implement, but the motivation for designing systems around them (the effectiveness of the index calculus on elliptic curve groups) and the discoveries made on attacking them (like the MOV attack that transforms ECDLP problems to FFDLP problems) are not at all simple.
Arguably, elliptic curve is an odder corner of mathematics than lattices.
100%. Another underappreciated point is that factoring connection is very misleading. That is, we don't know if factoring is equivalent to breaking RSA (in particular, the algorithms breaking low-exponent RSA (which one shouldn't use for a number of reasons) don't factor the modulus). It might be true that computing e-th roots (i.e. breaking RSA) is tractable while factoring is not.
Similarly, PKCS is complex but every part of PKCS is there because without it there is a concrete attack. Burt Kaliski (former chief scientist of RSA Labs) has an amazing talk which goes into detail about this: https://www.youtube.com/watch?v=sqsDKjPaJVg For example, why does RSA need randomized padding, besides the trivial IND-CPA violation? Because if you encrypt the same message to many recipients, the attacker can use Hastad's attack, which uses lattices in a deep way, to recover the message from non-randomized ciphertexts https://en.wikipedia.org/wiki/Coppersmith%27s_attack?useskin... Very much like an airline checklist in a way :-)
Another nugget in the talk: how RSA embedded a public key in their products and bootstrapped VeriSign! Can't recommend it highly enough.
Elliptic curve is not at all simple to comprehend! It's easy to implement, but the motivation for designing systems around them (the effectiveness of the index calculus on elliptic curve groups) and the discoveries made on attacking them (like the MOV attack that transforms ECDLP problems to FFDLP problems) are not at all simple.
Arguably, elliptic curve is an odder corner of mathematics than lattices.