Compared to just deploying totp normally as a PAM module this is a horrible idea. Much harder to ratelimit, much cheaper for the attacker to bruteforce codes.
Doesnt the TOTP in this case concern the listening port for sshd, so it doesnt actually touch the authentication in any way? Just switches the port in a TOTP’esque manner
The point is that if the attacker somehold got hold of the primary login credentials (username + key/password), then they can easily bypass this scheme with a port/address scan. This can be done very quick[1] and is hard to rate limit. Furthermore, an attacker that can eavesdrop on the user's connections can infer the OTP since it's being transmitted in the open., but if it was done through a PAM module they wouldn't be able to because it's encrypted.
1 million IPs per second is doable with gigE if we're just sending a syn to port 22. I'm going to go out on a limb and assume your server has at least gigE.