I found third party CDNs were almost always the cause when my personal site felt slow. My personal server responds quickly every time, because it's not subject to the same heavy traffic fluctuations that CDNs are.

To that end, I serve all the font files from the server and run a self-hosted analytics tool. Most Google fonts are just conveniently hosted fonts from third-party foundries, so it's not like you are supporting Google by downloading them.