The slang phrase in question may well have been "Pull your finger out", meaning, "Get on with it." It goes with the index-finger-through-closed-fist gesture.

A generation later (1940-45), the expression "finger trouble" meant hesitation (at best) or cowardice (at worst).