I'm also from a small southern college town and the housing market has been absolutely absurd for the past decade or so. It's always seemed like football fans looking for vacation homes and retirees were buoying it.
I believe they play a part here as well. My own home has increased in value ~$160k in 3 years. We’re ready to leave this town. The college kids and the retirees make it an absolute chore to live here.
