"...maybe from red pom-poms worn on British naval uniform hats..."

Far be it for me to contradict an organ of verity such as Wikipedia, but to my certain knowledge the Royal Navy has never had red (or indeed any other colour) pompoms on their caps (never hats). That particular fashion statement belongs to the French navy.

Another theory, although there's a school of thought against it, is that it derives from POM — property of (her/his) majesty — which was printed on the back of the convicts clothing.