If it were not so there would have been no need to Boris Johnson to apologise,

He only apologised for any offence caused, though it caused none at the time (2003)
He did not acknowledge or accept that the word is racist.

if it were not so Enoch Powell would not have been castigated for using the word.

He was not castigated for using the word.
The word has no derogatory connotations where our black communities originated.
If they are not offended by it, why should anyone else be.