I have always taken it to mean "Bollocks".

In the army if the tune of Colonel Bogey came up instead of singing "Hitler has only got one ball" that phrase was replaced with "Bollocks and the same to you".

The V sign that is reverse way to that used by Churchill.