I used to sing the song, and I thought it obvious that the narrator was a sailor who had fallen for a girl who lived in a forested area. The locals were saying that she wasn't for him and his lifestyle, and he was reluctantly agreeing.