The traditional gesture used to be the balled fist with the thumb poking out between the first and second fingers.
It used to be called "pulling a fig" or "giving the figgoe" and in the 17th century was the equivalent of the middle digit extended (or, in the UK, the 'V' sign with knuckles outwards) and was clearly a gesture meaning "Fuck off!". Hence the expression "I don't give a fig for...".