I grew up in the segregated south. So my comments are based on my personal experience rather than research or someone's written interpretation.
Most of the poor people I knew (black or white) were what some might consider working class poor and being so meant that most of their time was consumed in their efforts to survive and provide for their families.
I suppose the theory that the poor need to look down on someone in order to make themselves feel better/superior may hold some truth but I don't believe it would be accurate as a "blanket statement". I've witnessed poor people, regardless of race, being far more willing to help each other in times of need and/or crisis. I've also found that the most giving people are often the people who have the least to give.
If you think slavery and racism were restricted to the south, it might serve you well to look a bit farther. I've witnessed racism in various other parts of the USA, often equal and sometimes even more intense than what I've seen in the south but I'm convinced that the worse part of racism and slavery in our history was the institutionalized variety by those who claim to be so opposed to it.

If you want to classify someone by their wealth or lack of, there will always be someone wealthier or poorer than you. If it's your objective to classify someone by their color, you'll always find someone a bit darker or even lighter than yourself. What I find interesting is the people who say it shouldn't make any difference seem to be the people who mention these differences the most (couldn't be that it's what they notice most.... could it?).

Ever lived in a home with no electricity or running water? Ever worked in the fields from sun up til sundown? Ever been criticized for wearing clothes that were not fashionable or they were even "handmade"? Ever live in a community/neighborhood that was predominantly white, predominantly black, predominantly hispanic, etc. while being the minority? Ever been shot at for speaking your mind?

There are some lessons and experiences in life that can't be put into words. Doesn't mean we shouldn't make the effort but until you've lived an experience it can be difficult to fully understand.