I feel like there was a Cambrian explosion of UI design around the period of Geocities and web 2.0 sites like Digg. The Web was still in the early days of being monetized, and it had a fraction of the investment it has in terms of dedicated design teams, and all of the a/b testing to identify optimal design patterns for driving various actions.

Today we have survival of the fittest, and non-commercial design largely mirrors it due to that being what the tools and libraries to make sites offer.