Harold Vincent Milligan wrote as early as 1920 in his Stephen Collins Foster: A Biography of America's Folk-Song Composer (G. Shirmer, 1920, p. 68) as follows:

There is a tradition that "My Old Kentucky Home" was written at the home of a relative of the Foster family, Judge Rowan of Bardstown, Kentucky, who was also U.S. Senator. This story cannot be verified. It was certainly not necessary for Stephen Foster to be actually in Kentucky at the time, any more than it was necessary for him to be familiar with Florida in mentioning the Swanee River. The important thing is that the song rings true and expresses an emotion deep-rooted in the human soul.

See also The Minstrel of the Alleghenies by Harvey Gaul (issued by Friends of Harvey Gaul, n.d. [1952*], pp. 54-56; Click here). He is skeptical about the story.

*This date is from Calvin Elliker, Stephen Collins Foster: A Guide to Research (Garland); "1934" is written on the cover & title page, but it is the year when Gaul wrote it.