I sing it as Seven Yellow Gypsies and the lyric would not scan properly if it were turned to travellers...I think it must have been a joke. I meet plenty of people who refer to themselves as "gypsies" and don't make a big fuss out of it.