I don't think that is the right way to think about it, it has more to do with trajectory bending. For example, let's say an object is going radially outwards away from the sun at 10 km/s when it encounters a planet that is orbiting at 30 km/s.
Let's say the trajectories are at right angles, so their relative speed difference before the encounter is 31.6 km/s.
I now claim that this relative speed difference is the same before and after the encounter, so there's no "less time spent decelerating on the way out". (This is an approximation, I'm pretending that the planet provides an inertial reference frame with conservation of energy. The centripetal/sunward acceleration is small enough for that.) Let me show how you can still get a gravity assist under this assumption.
If the object passes "in front" of the planet in such a way that its trajectory gets bent towards the retrogade direction of the planet, then its speed becomes 30 - 31.6 = -1.6 km/s. So 1.6 km/s relative to the sun, and direction retrograde of the planet.
If the object passes "behind" the planet in such a way that its trajectory gets bent towards the prograde direction of the planet, then its speed becomes 30 + 31.6 = 30 + 31.6 = 61.6 km/s relative to the sun, this time in the prograde direction of the planet.
I hope I've convinced you that gravity assists work by taking the relative speed difference, and reorienting it.
Let's say the trajectories are at right angles, so their relative speed difference before the encounter is 31.6 km/s.
I now claim that this relative speed difference is the same before and after the encounter, so there's no "less time spent decelerating on the way out". (This is an approximation, I'm pretending that the planet provides an inertial reference frame with conservation of energy. The centripetal/sunward acceleration is small enough for that.) Let me show how you can still get a gravity assist under this assumption.
If the object passes "in front" of the planet in such a way that its trajectory gets bent towards the retrogade direction of the planet, then its speed becomes 30 - 31.6 = -1.6 km/s. So 1.6 km/s relative to the sun, and direction retrograde of the planet.
If the object passes "behind" the planet in such a way that its trajectory gets bent towards the prograde direction of the planet, then its speed becomes 30 + 31.6 = 30 + 31.6 = 61.6 km/s relative to the sun, this time in the prograde direction of the planet.
I hope I've convinced you that gravity assists work by taking the relative speed difference, and reorienting it.