"Only a small minority of Americans ever benefited from slavery (mainly New England merchants and large Southern landholders). Most of the time, most Americans have had to do their share of the 'dirty work'. "

You are correct in identifying the groups who benefitted most directly and to the gratest degree from slavery, but I do believe that the entire free population of the US has always enjoyed a degree of economic comfort, a "safety net," if you will, as a result of the severe economic exploitation of black people ~ which did not end with the Emancipation Proclamation, but continued through the Jim-Crow/sharecropping era (which may well have been worse than slavery from an economic point of view, if not in terms of human dignity).

I'm not surprised to encounter objections, disbelief, etc., because the insidious thing about this phenomenon is that almost everyone has always been blissfully unaware of it, which is exactly why otherwise thoughtful folks are so easily able to identify with the privileged few, and to act on the assumption that what's good for the fattest of the fat cats is good for themselves as well.