> Glass is hardly the problem there - if the conversation is interesting, people will turn off any music being played and ignore whatever is happening on Glass.

Not necessarily. People can be very distracted by cell phones which are normally kept in one's pocket or purse. Having glass on your face with visual pop-ups would be even more distracting, regardless of the conversation, because it would be a lot harder to ignore incoming alerts or other cues.