M,

The reference is to Marryat's _A Diary in America (Vol. 1_, the London edition, 1839. Please see page 40, here:
https://books.google.com/books?id=hK46AAAAcAAJ

Note again that Marryat was seeing this patent windlass in use for the first time, as well hearing the song, "Sally Brown" for the first time and, implying through is language, that it was generally musically different than songs he might have heard before.

In my interpretation, there is a correlation between the use of the patent windlass and the use of this style of song.

The earlier "windlass" was operated in a way that was not conducive to singing the "chanty" type songs. A close reading of multiple sources in which describe the earlier handspike windlass compared to sources that describe the patent windlass reveal a difference in the sort of vocalization (if any) at work. The latter day category of "windlass chanties" has led to conflation, even worse when all is subsumed under "heaving chanties" and a worse situation still applies when one interprets the truism "chanties were not strictly assigned to any one task" too liberally.

I don't believe Melville was acquainted with the patent windlass during his seafaring experience. It was not yet well established technology in sailing vessels then. Later, the patent windlass became an essential tool in whaling vessels, used to haul up parts of whales (as was the handspike windlass). The absence of the patent windlass in Melville's otherwise rich description of shipboard life is, I believe, in accord with the absence of "windlass chanties."

If this reading is accurate, it suggests:
1) the "booble alley" of Melville was a line in some song, unknown to us, that Melville experienced sung at the handspike windlass -- and about which we may never known, but which was probably not like the familiar chanties. These utterances at the handspike windlass (generally speaking) were so choppy and, to invert Marryat's observation, un-musical, that they do not cohere into well-defined "songs," but rather consisted of assorted phrases.
2) Melville read Marryat and, though he was previously unfamiliar with Marryat's "Sally Brown" (perhaps reflected by Melville's non-specific phrasing, "some chorus...") he took the liberty of setting this "windlass" song to his own windlass in his fictional work.
3) something else ...! But there are several strikes against the probability that John Short's rendition of "Haul Away..." can be matched to what Melville had in mind. The Folk Revival phenomenon has reified the John Short rendition.

Gibb