It can be that slow, yes. There is no assurances that will be fast, you're just hoping it will be.
Now you could argue "that's true of anything!" but in general you are forced to depend on local IO/CPU/RAM otherwise your site is struggling in general, by adding DNS/SMTP/etc you're just adding more "stuff" that your site relies upon.
Plus SMTP servers are "allowed" to go offline for short periods. The client should try to resend the email 2-4 additional times before giving up. Clearly a PHP script cannot do that.
Thanks, I understand. Configuring PHP that way would be an unmitigated disaster. If you want that functionality, it makes the most sense to configure Exim or whatever to do that for you.
Just realized I edited out (before I posted) my qualifcation for my statement, but it should've been prefaced as "from what I've noticed with no hard numbers."
The reason I assume that it has the feeling of being slow, from the applications I've had to clean up, is because many times there's processing happening right before the mail() function. So while, yes, its relatively fast the the total time is what feels slow.
And for comparison I've ran your commandline test as well with two of my cheap-o VPS instances, one of which is a Linode instance: