A criminal cannot walk through a room without leaving abundant evidence of his (or her) presence in DNA, fibers, characteristic dirt grains (remember OJ?), insect eggs, etc. Aliens could not have done anything more than a fly-by without leaving an abundance of physical evidence. Everything found so far has a rather prosaic explanation. Doesn't mean they're not "out there" (statistically speaking in an infinite universe there's a 100% chance that they are, but I suspect they are too far away to visit)