In any given exchange, the speaker is likely optimizing both honesty and diplomacy, trading one against the other as the situation requires. The phrase "to be honest" signals that the next bit is going to sacrifice an unusually large amount diplomacy for an unusually large amount of honesty. Or something.

It's a verbal flag and a hedging move. "I am about to say something potentially hurtful, but I am doing it with your best interests at heart, so don't instinctively retaliate" is kind of what it's used for. Reading too deeply into the literal meaning of the words doesn't really go anywhere, any more than with other verbal rituals. Compare to the greeting "How's it going?", often used even when the asker doesn't really want to know how it is going.

I resisted posting my pet peeve because it's not really the point of the OP, but now I can't help it. Let me explain...

http://www.hnsearch.com/search#request/all&q=%22let+me+e...

The fact that there is more text in a comment let's the reader know more explainin' is on the way. No need to point out that you're about to elaborate. Are people worried we'll get to the end of their paragraph and not know if the next paragraph is related?

It makes sense in face-to-face interactions, particularly when addressing are group. Sometimes, if you say something controversial, people will respond dismissively before you've had a chance to justify yourself. "Let me explain" is used as a way to reclaim the floor, so to speak.

For better or worse, people are transferring this pattern to written language as stylistic device.

I think "to be honest" is a euphemism for "to be blunt".

To the extent that people usually politely self-censor, yes, they're usually not being honest with you.

I agree, I prefer "to be frank" as I think it avoids the misunderstanding.

Would you believe me if I said yes? :)