The most times I've seen an English flag flying has been on C of E churches when they are having a fete. Not that threatening.
All right, there are football rowdies who use it these days - but even so, they are more likely to use the Union Jack (and I know "Union Flag"). And that (unfairly enough) is the one with most of the nasty associations. That's the one with all the association with Nazis and and "Loyalists" and Tories.
To avoid that kind of stuff anyway, the day should be celebrated with all the relevant national flags - so as well as England, that's Greece, Germany, Georgia(the European Georgia that is), Canada, Catalonia, Lithuania, Palestine and Portugal. Which should make for a very lively display.