How would that happen, exactly? If you were to have my bank account number you could give me money, but you couldn't just take it out. It's not like a credit card number.
Have you never had anything billed to your bank account? You give them your routing number and account number, and they take the money directly. Some sites will make you prove it's your bank account by asking you for the amount of two small deposits, but many don't.
This isn't hypothetical. It happens.
> In an attempt to prove that the public furore over the 2007 UK child benefit data scandal was unjustified, [Jeremy Clarkson] published his own bank account number and sort code, together with instructions on how to find out his address, in The Sun newspaper, expecting nobody to be able to remove money from his account. He later discovered that someone had set up a monthly direct debit for £500 to Diabetes UK.
