The thing about free speech is that everyone has their own criteria of what should or should not be included into it.
When someone talks about "free speech absolutist", one expects they will include everything. No exceptions. After all, that's what the "absolutist" is supposed to be doing here.
If you want to add exceptions to free speech, then you're not an absolutist. You just have the same lame relativist free speech definition as everyone else. Except your criteria is different than the others. Which, I mean, no problem, but at least be honest.