Doesn't that assume that an infinite amount of time has already passed? Which seems like a contradiction in terms.

I don't see how it would be a contradiction in terms; we can easily speak of the possibility of an infinite amount of time in the future, why not also in the past?

(It may be factually wrong, but that's not a contradiction in terms).