Symbols symbolise what they are intended to symbolise. Treating them as if they had some power in themselves, regardless of the intention of the people who use them, or in the case of physical objects, who made them, is giving in to a way of thinking that has done enormous damage in the past.

The symbols the Nazis used, swastikas and salutes and so forth, were symbols that had been around for thousands of years, and they used them for evil purposes. The same way they used the German language for evil purposes. There's really no more sense in suggesting that all the symbols they used should be forever banished from human use than suggesting that the German language should be similarly banished.