Guitars can produce mellow, smooth, seductive sounds; hot-blooded, earthy, humpin' sounds; sweet, innocent, lilting sounds; etc.

Banjos can produce energetic, jangling, intrusive sounds.

Yes, that's stereotyping; yes, there are exceptions; yes, not everyone sees it that way. But among those who do find a guitar sexy and a banjo not sexy (who I believe form a sizable proportion, likely a majority, of people), I'll bet that's their perception.