To my ears, "rough and tough and hard to bluff and used to many hardships."
rings the sorts of bells that the chorus to Barnacle Bill the sailor also rings;
"I'm old and rough and dirty and tough," cried Barnacle Bill the sailor.

If one were trying to entertain polite company and avoid the earthy activities of Barnacle Bill, Joe's phrase would just about scan similarly and also allow the same accumulative narrative.

Cheers, Rowan