Jack the Sailor,

You said "there may have been a small fringe of white supremacists who were anti-slavery. They were rare exceptions".

I wish you were right, but in fact it was not a "small fringe" of people who supported white supremacy and who also opposed slavery. Lincoln himself was a white supremacist, and said he had never met anyone who was NOT a white supremacist.   

Below are Lincoln's own words on the subject, from the Fourth Lincoln-Douglas debate. Notice especially the last line. Read that line and then see if you don't find it inspiring that Tim Scott was elected to the U.S. House of Representives, with 66% of the vote(!), from the district that includes Fort Sumpter!!!
   
Mr. Lincoln's Speech

(September 18, 1858)


LADIES AND GENTLEMEN: It will be very difficult for an audience so large as this to hear distinctly what a speaker says, and consequently it is important that as profound silence be preserved as possible.   1
While I was at the hotel to-day, an elderly gentleman called upon me to know whether I was really in favor of producing perfect equality between the negroes and white people. While I had not proposed to myself on this occasion to say much on that subject, yet as the question was asked me, I thought I would occupy perhaps five minutes in saying something in regard to it. I will say then that I am not, nor ever have been, in favor of bringing about in any way the social and political equality of the white and black races, that I am not, nor ever have been, in favor of making voters or jurors of negroes, nor of qualifying them to hold office, nor to intermarry with white people; and I will say in addition to this that there is a physical difference between the white and black races which I believe will forever forbid the two races living together on terms of social and political equality. And inasmuch as they cannot so live, while they do remain together there must be the position of superior and inferior, and I as much as any other man am in favor of having the superior position assigned to the white race. I say upon this occasion I do not perceive that because the white man is to have the superior position the negro should be denied everything. I do not understand that because I do not want a negro woman for a slave I must necessarily want her for a wife. My understanding is that I can just let her alone. I am now in my fiftieth year, and I certainly never had a black woman for either a slave or a wife. So it seems to me quite possible for us to get along without making either slaves or wives of negroes. I will add to this that I have never seen, to my knowledge, a man, woman, or child who was in favor of producing a perfect equality, social and political, between negroes and white men.