No, I agree, I don't know what the general concensus is, and I agree we should tread carefully. However, I likewise think it is patronising to assume how others feel without asking them directly - that's the way we treat children & those who we assume incapable of expressing themselves.
I have plenty of friends of differing ethnicity & none of them have ever found it racist, indeed most of them assume it has something to do with paganism & being gothic & weird. I am bewildered by anyone whose mind works in such a way that it can look at a modern day morris dancer & somehow imagine it resembles the dress & behaviour of a black person.