I told him that the general reaction to an Engishman saying such a thing was to label him a racist or BNP member and he seemed not fully in agreement.

I wouldn't be in agreement either. In fact I'd be rather suspicious that somebody making a comment like that might well be a bit of a racist. (And in your case I know that would be completely wrong.)

I really don't think that many people in fact do see fondness for Morris dancing and suchlike as an indication of racism.