I think the issue mentally is if you treat the gravity of a planet like a valley, where you accelerate into it but on the way back out you lose all that gained velocity. The difference here is that the valley moves in the opposite direction ever so slightly as you pass it, and you gain the momentum from that valley's movement. At least, I think that's how it works.