Just realized I edited out (before I posted) my qualifcation for my statement, but it should've been prefaced as "from what I've noticed with no hard numbers."
The reason I assume that it has the feeling of being slow, from the applications I've had to clean up, is because many times there's processing happening right before the mail() function. So while, yes, its relatively fast the the total time is what feels slow.
And for comparison I've ran your commandline test as well with two of my cheap-o VPS instances, one of which is a Linode instance:
