That's the idea with the expiry date, and the CVV, and the zip code. The problem is, it doesn't seem possible to convince businesses not to hold on to whatever security info is required to charge the card in plain text, so whatever the relevant details are inevitably get leaked from some hotel or eCommerce giant that really shouldn't have them in the first place, but hasn't set up a way to securely verify credentials with the bank without literally recording them.
You can keep adding on additional pieces of bullshit information customers need to remember all you want, none of it will matter as long as banks and credit card companies don't force businesses to treat them as actually sensitive information.
I think that enforcing what you're suggesting is incredibly hard and I don't think can scale, it's what PCI-DSS and similar are meant to tackle, it really doesn't work in my experience.
This is a protocol/product problem, it's wild that to make a payment all the crown jewels need to be put on the wire. It's about time that payment devices and the whole ecosystem adopts some sensible cryptography that, at minimum allows signing payment requests, and ideally keeps its keys private.
Although this whole problem is kind of already solved by 3DS2, albeit not in a great way.
You can keep adding on additional pieces of bullshit information customers need to remember all you want, none of it will matter as long as banks and credit card companies don't force businesses to treat them as actually sensitive information.