Here you go, Kevin (and anyone else interested) try this for a start:

Anbinder, Tyler G.:Nativism and Slavery: The Northern Know Nothings and the Politics of the 1850s. Oxford University Press, USA 1994

Plenty more where that came from.