For one thing, examining referer is a common way that a server determines a request is not a hotlink. Sure you can do something more complicated with cookies or whatever, but lots of sites are just using referer and they'll break if the client doesn't send it.
Well, it'd mean that any site blocking hotlinking would also automatically block direct bookmarks/URL entry, too, which isn't really in the "interest of the user" either, I'd say.
