Simple answer: Imagine time travel is possible. OK?
Now, think about the absence of tourists from the future. Doooh... Seen any (aside from Charlie Sheen)
That's quite an argument against the existence of time travel - simplified, sure, but quite reasonable and
still a variant of 'Fermi Paradox'
Time machines can never go back further than the moment of their creation. Simple.