Depending on your concept of time, it very much is. If you choose the same abstraction of time for reasoning about the program state and the real world, the same "reference" might change its properties over time. The speed of a car is mutable over time. The question is if the way we reason about time when talking about the "real" world is appropriate for reasoning about time in programs.

Attach a date stamp to your references and facts and BOOM, everything is immutable!

Yes, you mention it. Time.

Time is a real property just like IO, effects, values and types. If people stop hand-waving, we would have fewer bugs.