I have not heard the song in quite a while so memory is a bit foggy but I had the impression Young Clary was a bully and drunk to boot and had pulled a knife when the narrator went to the defense of his wife. I assumed the neighbors were afraid of the bully but it could have been the narrator's politics or a bit of both.