Once you decide to leave and take a new job your allegiance is now to your future employer. My only regret in leaving jobs has been not taking all my holidays and staying too long after deciding to leave. If you do feel bad about leaving then c.f. Stockholm Syndrome.