You shouldn't need to care about that; video formats should support clipping to non-keyframes by including the preceding GOP at negative playback times, so it skips to the frame you started the clip at.
(MOV in fact does support this, but ffmpeg doesn't play it back properly.)
(MOV in fact does support this, but ffmpeg doesn't play it back properly.)