Wow, this got interesting fast...

My origin was when I was taking a linguistic field methods class, we had an informant who spoke a Bantu language, and we would ask for words for Mother, father etc, learning to be field linguists. Whenever we referred to Bantu languages he would look upset, and finally he stood up (he was seated facing us) and said Why do you keep calling these languages Bantu languages? And the professor said, because that is the term for this family of languages. He was very, very upset, and said that that was terrible because Bantu is an insult in all three of the languages in that family that he spoke, that as far as he knew it was an insult in all of them, and how dare the linguists use that term. We were all very, very apologetic and never used the term Bantu again within that class, using instead the KiSwahili family of languages.

I still think that some linguist early one asked some informant what was being spoken by a neighboring tribe and was told "some Bantu language" - meaning some )*&%)%) language - but the linguist thought it was the word for that family of language.

Reminds me of the fact that something very close to "abidjan" in one of the languages spoken near Abidjan in Ivory Coast where I grew up means "Go away, I'm fishing" - apparently the answer when the explorer asked What is that town over there!