I was just off poking around for confirmation it was Andrew Johnson (as stated above) who had a black mistress. My memory said "Andrew Jackson," but it's often wrong. I still have some checking to do on that score, but I did come across this quote in a site dedicated to U.S. presidents' views of blacks.

   

    Abraham Lincoln
    "I am not nor ever have been in favor of making voters or jurors of Negroes, nor of qualifying them to hold office, not to intermarry with white people; and I say in addition to this that there is a difference between the white and black races which I believe will forever forbid the two races living together in terms of social and political equality. And inasmuch as they cannot so live, while they do remain together there must be a position of superior and inferior and as much as any other man am in favor of having the superior position assigned to the white race.

cheers,

david