My experience matches yours. I found AmEx accepted maybe 50% of the time, if not less. But Visa or MasterCard were accepted 99% of the time, both big and small businesses.

I suspect you're right: it depends on the specific card/bank, not whether it's credit or debit.