In Britain, it is almost never used by private citizens except as a fascist emblem, and non-white people in Britain have every reason to react to it as Jews do the swastika - it stands for being spat on and beaten in the street, gang attacks and firebombs.
Fascist like the new Mini, the dresses the Spice Girls wore, the Last Night of the Proms, the flag that Lewis Hamilton and Amir Khan draped around their shoulders...?
Maybe for a while in the Seventies and early Eighties the Union flag was hijacked by the NF and other Far Right organisations, but I don't think it's got that connotation now.