I have been doing a bit of an overhaul on my notes/references to this song. A few points are of interest.

The totality of documented versions make it clear, I think, that "blow" meant "knock," "hit," etc., and the action would be done *by* a person. The "man" to whom the action was done may have been an actual person, however *I* believe it may have been metaphorical, where "man" stands for an item of cargo to be stowed in the hold.

There is not enough evidence, to my mind, to support the claim of some 20th c. casual writers that this chanty had a special significance for the Black Ball Line. There was indeed a version sung with a clever narrative about a Black Ball ship, but that's all. And whereas it's possible that for some people the word "blow" took on the idea of getting beat on by the ship's officers, it looks unlikely that that was the original sense. It's the workers singing who are giving the blows.