I think if it's possible to change the ping algo to reduce the potential for wall clock change affecting the measurement.

For example, instead of using SO_TIMESTAMP to measure wall clock diffed with the send time, maybe it's possible to grab the wall clock just before recv() and use wall clock to measure the event-loop time only (which is way shorter than the ping rtt).