Yes, it is so hard. I am learning Hindi and I can't even hear the difference between several of the consonants, making it very difficult to measure how well I'm pronouncing them. My Chicago friends can't hear the difference when I say Kyle and Karl, so can't teach themselves to imitate it. And even if they could, changing pronunciation habits takes a lot of attention to the way you speak, which takes effort, which is basically the definition of hard. Its nice that its easy for you, but you should try thinking of it as a skill you have rather than sound like you think everyone else is just lazy.
Different English accents have a massive variety of vowel and consonant sounds, and usually one can find a "difficult" vowel sound in some English accent.
As a counter-example, in my experience Spanish accents have very little vowel sound variation.
Perhaps learning how to mimic accents of people from different regions (Irish, Jordie, Scottish, Alabama, BBC etc etc) of your own mother tongue gives you opportunity to learn how to hear and say different vowels and consonants from another language. Or practice the accent off non-native speakers e.g. a French accent for a Parisian speaking English.
I didn't mean to say that everyone else is lazy - I just thought that having a native accent wasn't a priority for most people. People can usually make themselves understood perfectly as long as their accent isn't too deviant, and having an accent has its own charm :)
