That’s not how a non-disparagement clause works.

It puts restrictions on what you’re allowed to say. It doesn’t require you to correct what other people say.

If your badly thought through assumption was correct, the logical response from him would be to simply say nothing.