Les, you were indeed right - we did divide on this; but then it was always going to happen, as your question only allowed two sides: Like it or hate it, just a song or symbolic of whatever, black or white.
The issue I find is not the division in the responses, but the asking of such a polarising question in the first place. After all, whose ethnic past is untainted?
Imperialism and exploitation, eh? Well, my ancestors were creating empires and "democracy" on the back of slaves when the favourite colour in the british Isles was woad-blue. My very hero, Alexander the Great, was guilty of that too. But he also did a lot of good; and while I may with the benefit of advanced civilisation and knowledge decry much of what he did, I will not judge him out of context and will admire him for the good deeds.
The reason I find the asking of the question disturbing in the first place is that it reminds me of the German society's post-war self-deprecation: so much over the top, that we are now reaping its fruits in neo-Nazism. Because we all need pride in our background. And we all deserve it. To strip a person of that is inviting a reaction and serves no purpose. And to do so on the basis of historic facts viewed with hindsight achieves precious little. Just like hating one's parents for their bad points doesn't make one better. We should learn from the past to improve the present and prepare the future - not wallow in it.
The question of whether singing "Rule Britannia" today is a good thing and whether it implies tacit approval of past practices only serves to muddy waters and divert from questions that are more relevant now: Like the various forms exploitation takes in our times, more insidious but just as despicable.
As an aside: I remember singing "RB" while watching Last Night of the Proms on TV in 1996, having at the time been away from the UK for a few years, brandishing sparklers out of an Amsterdam 9th floor window like a demented fool. I confess to a couple of tears at the time. Heck, I'm not even a Brit! But I lived here for many years, have several good mates, and the tears were perhaps because I was missing them, and all the other good people of this land. The song was just a symbol to me. And in 2000, back in the UK, I watched it again, and that time I took more notice of the ponces in the front row and got discusted and switched off. Same event, same song, same person, different reaction.