Yup. IMO not using a passphrase is nuts, considering there is practically no convenience loss (ssh-agent, OS X keychain, pageant) and fairly significant benefit. Protects against:
- accidentally exposing the private key (it happens, accidental wildcard in the shell), and
- attacks that aren't after your keys specifically, e.g. your home directory gets stolen.
In general I agree with you, with the caveat that there are a number of products that don't give a toss about security (lookin' at you, Hashicorp) that actually and unironically require you to have your keys unencrypted on disk.
I have since discarded these products (and others should too), but they do exist.
OTOH, it's defense in depth, and there are some vulnerabilities which could allow to exfiltrate keys without non-sandboxed-RCE
https://blog.mozilla.org/security/2015/08/06/firefox-exploit...
I reckon that a passphrase protected key would be safe from this attack (modulo offline brute-forcing)