You can surround your custom port by a couple of ports on which a simple server listens for connection attempts. Any connection attempt is considered hostile and the ip will then be blacklisted in iptables. This prevents portscans from reaching your port.
Different quality of locks in the ever-escalating arms race. Probably there are many many more sequential scanners out there. For the persistent actors who are doing random ordering or shuffle then you could add port-knocking for the real sshd... but then they just have to find a working client and sniff the connection requests... to which you add a TOTP step for determining which ports to use, and so on...
Excuse the old school metaphor - you put a lock on your door so your house is harder to break into, not to prevent anyone from breaking into your house.
Absolutely agree, when I wrote this I was thinking more of defending against the low hanging fruit - mass scanners.
Once someone has deemed you a worthwhile target and is carefully proving all ports, these more nuanced approaches become more worthwhile. Even then, a sophisticated adversary may have many unique src IPs at their disposal.
