One flaw in that argument is that it appears that the number of people who don't vote at all, either in primaries or in the actual election, is much higher than the number of people who vote for either of the main parties.

The other flaw is that, it assumes that getting elected is what it's all about. In reality, running in elections, for minority parties, is primarily an organisational and recruiting mechanism. Not a particularly effective mechanism, and not one I'd ever want to put much energy into, but that's another matter.