I thought the same thing reading this! I'm currently reading Prisoner's Dilemma by William Poundstone, which details the life of John von Neumann. According to his brother Nicholas, von Neumann likely got the idea to use punch cards for computers from the Jacquard loom. One of his father's (Maxwell von Neumann) clients at the turn of the century was a Jacquard loom manufacturer, and Max would evidently discuss his work with his kids everyday over supper.
Ah, you're right. Babbage's Analytic Engine was designed to use punch cards, and I totally forgot about the Hollerith Machine that I read about just last year. I wonder what other false information Poundstone's book has. Thanks for the heads up!
Not only did Babbage's Analytical Engine use punched cards, it was specifically inspired by the Jacquard Loom, as Babbage was a massive fan of it (and owned a portrait made on one!)
