One more quotation from a recent book, Ken Emerson's Doo-dah!: Stephen Foster and the Rise of American Popular Culture (Da Capo, 1998, p. 190):

It's entirely possible that Stephen Foster visited Bardstown sometime in his life and impossible to prove that he never set foot in Federal Hill. But the burden of proof falls on Bardstown's boosters, and all the facts point to Uncle Tom's Cabin rather than Federal Hill as the inspiration of "My Old Kentucky Home."

~Masato