According to[1], the backdoor introduces a much larger slowdown, without backdoor: 0m0.299s, with backdoor: 0m0.807s. I'm not sure exactly why the slowdown is so large.
The effect of the slowdown on the total handshake time wouldn't work well for detection, since without a baseline you can't tell if it's slow due to the backdoor, or due to high network latency or a slow/busy CPU. The relative timing of different steps in the TCP and SSH handshakes on the other hand should work, since the backdoor should only affect one/some steps (RSA verification), while others remain unaffected (e.g. the TCP handshake).
[1] https://www.openwall.com/lists/oss-security/2024/03/29/4