Well, Allen, while I understand the spirit of what you are saying,
I can't agree with you.

Prior to the end of slavery in the USA, free or freed African Americans in those times who lived in Southern states or in border Northern or Midwestern states like parts of Ohio or Pennsylvania lived daily with the threat of being kidnapped and sold as slaves.
So the legal institution of chattel slavery was a significant difference that had to have contributed to a greater or lesser degree to the behavior, thoughts, and psyche of both Blacks and Whites.

Not to mention laws that regulated the behavior of Black people including free or freed African Americans. For instance, the testimony of an African American was not accepted in court against a White person. And some states didn't allow free or freed African Americans to live there at all.

But if your point was that most Southern Whites were poor and didn't own slaves, the records show that to be true.

And there was secular musical interaction between the two groups.
To site one example, African American musicians were hired out to White 'frolics' and were noted callers at those barn dances.