People don't have an expectation of hard drives being removable(most people probably don't even realize they can be removed), whereas SD cards are explicitly designed to be mobile. People treat them as mini USB drives, so this behaviour is directly counter to most people's intuition, regardless of its technical merits.
What are the numbers of people swapping their SD cards out of their phone and into something else (and expecting it to work)? I don't know the statistics, but I'm willing to bet it's low. In fact, the number of people in general putting SD cards into their phones is likely to be very low, since a lot of phones don't support it.
I really don't understand your argument that phones just shouldn't support SD cards rather than having the option to expand the storage for cheaper than buying the next model up just to avoid user confusion. If every phone locked out useful features just for the sake of avoiding user confusion, Android wouldn't exist.
