The vessels mentioned in The Complaynt of Scotland are armed merchant rowing galleys. When the modern genre label gets rolled back to the 16th century, the military and merchant histories & practices become one and the same.

The rowing fleet the U.S. Navy put to sea at the start of the 19th century wasn't all that different.

If your personal, modern definition of “chantey” is a cappella merchant marine, the R.N. is excluded by the class of practitioner, not the practice itself.

If your modern definition allows for both the musical accompaniment of singing and the pure instrumental, the R.N. is a maybe or maybe not for “chantey” according to each consumer.

If you use a period dictionary you will not find “chantey”... period. Your ancestor's word(s) however, would include the R.N. practices by definition.