'blow' has lots of connotations and a wide range of meaning when applied to people. 'Look what the wind blew in!' as a newby or someone not seen for a while comes into the tavern. Is the line 'Blow him right back into Liverpool Town' present in early versions? 'Down' also can have lots of meanings in nautical parlance. It occurs just as a geographical direction in a lot of chanties and general nautical speech (Downeaster), so 'Blow the man down' quite possibly does not refer to violence of any sort. Even if you put the 2 words together they don't always refer to violence (knock-down prices for example).