The difference is that I can know with significant certainty that shoving my house key in a random lock won't copy the form of my key and send it to a 3D printer where a thief will get it and use it to access my house.
How can I know that won't happen when I use my USB dongle on a random coffee shop public computer?
You can actually safely plug U2F tokens into random computers and rest assured that the keys inside cannot be cloned. This provides security guarantees above and beyond that of physical keys! In fact, an untrusted computer can't* even MITM the authentication process of U2F, unlike with SMS codes! (Of course, an untrusted computer could fake the UI to try and exfiltrate other info from you, but that's beyond the scope of authentication itself.)
The worst a malicious client can do to a U2F token is to fry it :-)
*: There's some asterisks here, but if you want to know the details check out the U2F spec or this https://www.yubico.com/blog/creating-unphishable-security-ke... for a more accessible explanation. Banking-grade U2F/similar tokens actually behave like hardware crypto wallets and will show the auth request metadata on an internal screen, in case the device you are plugging into is completely untrustworthy.
A random lock (or a street-level camera FWIW [1]) would be better positioned to extract enough data to reproduce the key.
U2F tokens are "Trusted Platform Modules" of sorts. The keys themselves are never visible to the devices you plug them in. They are capable of answering challenge/responses, and having the URL part of the challenge prevents phishing[2].
I am much more comfortable not typing any password on a public computer.
How is that different from a password? How can you know that when you type your password into a random coffee shop public computer that the computer isn't running a keylogger and sending your credentials to criminals?
How can I know that won't happen when I use my USB dongle on a random coffee shop public computer?