I believe the song refers to the huge abattoirs in Chicago in the 19th and earlier 20th century when vast flocks of cattle were brought in by train from the Mid-West States. These slaughterhouses were largely staffed by black workers, many of whom had made the journey north from the poverty and discrimination of the southern states.
I have heard other explanations of its origins - for example that "Hard Times, Killing Floor" was a general name given to the Great Depression of the 1930s or that it refers to a specific killing of a woman by a man.
Certainly the words that Al quotes seem to refer generally to the depression, but for me the verse that clinches what it is referring to is the one that refers to the ghastly conditions and low status of the work. -

If I ever get off this killin' floor
I'll never get down this low no more
No-no, no-no, I'll never get down this low no more.