What seems clever to me is the super cool hack of getting sensor data while the video data is captured, then training a model on realtime video and backward time-shifted sensor data; result is a network that can predict sensor data 30 seconds ahead of time with visual data. This is then used to feed the walking model. That's the sort of super 'dumb' thing ML folks can do now; that task would have taken many many person years of work before, now, I speculate thinking of the idea to testing a network would be under a week. Verry cool.
I often think of this as the corollary to the bitter lesson: it is enough just to break some particular relationship between two things, and then throw compute at it. In the process of trying to recover that relationship, the model learns about the underlying real relationship - sometimes it even learns about each thing itself.
I think we're saying the same thing. Training pair at t0 is video_t0, sensor_t30. So we have shifted sensor data back 30 seconds, or put your way, we look to sensor data 30 seconds in the future.
