'change that fact' can be taken, in this context, to be an abbreviation for 'change the degree to which this fact is relevant to the given discussion'.

...but I'm sure you already knew that.

> your lack of objectivity

You don't sound particularly objective either. But perhaps you never claimed to be. I certainly didn't.