Hang on... I might check the date on that Harry Belafonte recording. Your hypothesis will only be plausible if the Belafonte version was recorded after 1958.
EDIT: I checked. According tothis site, it was recorded in 1957. Louise Bennett's album "Jamaican Folk Songs" was released in 1954, and Edric Connor, the Trinidadian actor and singer, also recorded it on his LP "Edric Connor Sings Songs From Jamaica", also the source of the earliest recording of "The Banana Boat Song". This looks like pretty conclusive evidence to me that the song is likely to be a Jamaican folk song.