Yeah, actually a fundamental rule of comedy: always get the audience on your side. If they don't know you, aren't rooting for you, or think you're on the "wrong side", they will misinterpret your joke as a real insult. Comedians directly insult people all the time and get laughs, but only because the audience believes they're on the same side as the comedian.