In my youth, I was told that $2 was the minimum bet at a racetrack, so gamblers tended to carry a lot of $2 bills. Gamblers were regarded as lowlifes, so if someone saw that you had a $2 bill, they'd tease you by saying, "Aha! I see you've been to the racetrack again."

Even if it's true that a minimum bet was $2, I seriously doubt that gamblers preferred to place bets with a $2 bill rather than two ones, or that there were more $2 bills in circulation at a racetrack than anywhere else. But that was the folklore, and if you didn't want to be stigmatized as a gambler, or were just tired of the jokes, you'd tend to avoid $2 bills.

I don't see why you'd be any more likely to mistake a $2 bill for a one, than to mistake a five or a ten for a one. In fact, it seems you'd be less likely to, because a $2 bill had some red ink, and no other bills did. Unless, of course, you were so poor that you never had any fives or tens.