60% of Americans enjoy watching football according to https://www.statista.com/statistics/1108544/american-footbal....
Even the 33% who strongly enjoy isn't a small minority. If a team decides to leave because the local city won't build them a stadium, it's very reasonable to think a majority of the local residents will be upset.