Think of them as words, not letters. It would supremely bother me if I were unable to write "colour" and always had to write "color".

I'm sure a Japanese person would be perplexed as to why that extra "u" matters since it's the "same damn word".