I use a sub-domain (e.g. @m.mydomain.com) for my catch-all and this hasn't happened to me. There are various easy ways for spammers to find out about domains, but sub-domains can remain relatively obscure.