Some past forms do not mean yesterday. They mean: this is not quite real, or it is late.
Distance, not last week
It's time we left. I'd rather you didn't. He talks as if he owned the place.
Should already be
The past form marks distance from the real now. Not a diary of last Tuesday.
It's time we leave. — loose
It's time we left. — the line we teach