As I try to wrap my brain around this could I assume then in order for this to work, on approach to the planet you have to be going in the same direction as its orbit?
Theoretically no, but if you're going the opposite direction the approach velocity is vastly higher and thus you need a (1) very massive planet and (2) much lower periapsis to have meaningful gravitational slingshot effect. In practice, between atmospheres (the big planets being Jupiter, Saturn, and Neptune), and (for non-gaseous planets) planets not being perfectly smooth spheres, that probably decreases the tolerances below any acceptable level in this solar system.
