I know exactly what you're talking about, nicheless, since I have the same problem--after much reflection, it has occurred to me that our problem is that when you move from niche to niche, you're alway playing relatively unfamiliar material with people who know it--that means that you are following and fudging while they are playing stuff that they have learned, practiced, and polished--

The real problem is that you are walking in to a musical situation cold, and expecting to be able pick up the repertoire, work out your parts, and create appropriate solos completely off the cuff--this only comfortable if:

A)Everyone else in the group is doing it too,
B)You are a more experienced musician than the people you are playing with, or
C)You are a hot shot studio musician--

The possible solutions are:

A)Organize and lead the sessions yourself
B)Find a group of people that like the kind of music you do
C)Get a bunch of less experienced players and teach them to play what you want
D)Polish your "Pick-up" musician skills--