My understanding is there are more fundamental reasons for why Go compiles faster, like how it handles "includes". And not supporting the C++ templating system.
And some things that slow down compiles are not to improve optimization. But to support complex, higher level language features. Go seems to hit a sweet spot of being highly expressive without slow compile times.
Well, it’s quite easy to be fast if you are just spewing out barely optimized machine code. Compilers aren’t slow just for the sake of it.