>I have already been getting spam through invites (where they get auto-added to my calendar even before I have a chance to see what they are).

Fortunately you can turn this off in the Google Calendar settings. This way they only get added to your calendar if you accept.