The problem with ignoring that they missed many states by 5-7 points is that it completely scrambled the view of down-ballot races. Democrats poured $300 million into Senate races the lost by large margins. Not only is it unlikely that Democrats gain the Senate, they lost seats in the House. That’s an insane result when national polls were predicting 7+ for Biden.
I’m adding to the point about why margins matter not contradicting Silver. (Though I’m convinced that Silver understands statistics better than politics.)