Heck, sites can be faster if they don't use AMP. Sure, most sites aren't, but it's certainly doable. AMP, aside from Google caching your site on the SRP, doesn't really do anything crazy when it comes to site performance.