I can imagine various possibilities, Gibb, none of them entirely persuasive.

1. Pronunciation in the 1840s to roughly '60s varied for one reason or another, perhaps because rationalizing, Frenchless speakers thought they heard "tchant" and "tchantay."

2. Nordhoff heard "shant(y)" but wrote "ch" to show etymology. He was thinking of "chevron" or "chevalier."

3. Or else his copy-editor, unfamiliar with the word and/or squinting at Nordhoff's handwriting, changed it. If Nordhoff wrote "chanty," the editor might have changed it as a slip of the pen. But then why keep "chanty-man"?

4. The word "shanty" was not yet in universal use. Nordhoff thought of the hauling songs as "chants" and the "chanty-man" as a "man of chants." Or else as the man who cried "Chantez!"

Concerning 1, several sailor writers specify that the "correct" pronunciation is "shanty." But how many say nothing -- perhaps indicating that they thought "tchanty" was just fine. Maybe the prescriptivists believed that the pronunciation should follow the eytmology.

That's all that occurs to me at the moment. As I say, none of these possibilities is fully convincing.

But one (or more) may just be right!