I've been that interviewee before and I was much the same. For me I realized it has to do with viewing the time spent in the interview as some kind of unspoken contract. You both felt good about the interview but you aren't accepting. Feels like you're letting them down.