can't display a country's flag because it might offend immigrants

Surely if you say that a black man will be offended by seeing his country's flag because his grandfather came from Jamaica you are implying that he has no right to be in the country because of his skin colour.