I think that the main indicator that people cannot and never will be able to travel backwards through time is that they haven't.

If they had, then we'd be inundated with time tourists observing major events, and we'd also be swamped by advertising from the future.

This might be because it's impossible, or it might be because mankind is toast. If ants or whales could travel through time, we wouldn't know.