There are so many aspects to this question that it evades simple analysis. My first thought was that any number works as long as everyone is working toward the same end. But then I thought of groups in which totally disparate elements came together to create something truly unique. One example of this is McKendree Spring, which used elements of folk, jazz and electronic music to create something truly unique.
If you're not familiar with them (check them out!!), the band was led by Fran McKendree on rhythmic acoustic guitar and dynamic vocals. I don't remember the names of the other members, but they had electric bass, electric guitar and electric violin. They used a drummer at times, but didn't really need one because McKendree's guitar playing was so strong, and so rhythmic. The electric guitarist ws very jazz influenced. The violinist played theremin and added the electronic edge. Everything worked in support of the singer and the song. Very cool, interesting approach and dynamics.

I guess all I can say is that the ideal size of the group or ensemble depends on what it takes to achieve the sound that you hear in your head. Personally, I see two kinds of groups - for lack of a better description, those in which sounds blend and those in which sounds collide. The strength of a blending group lies in how each individual sound compliments the other. This type of group needs to be able to point every individual in the same direction, working toward a single sound. In a lot of cases, the members of this type of group don't need to be virtuosos; simple individual parts often work better in this context than complex ones.
On the other end of the spectrum are the groups that find strength in the clashing of elements and the resulting tension in the interplay of voices and instruments. The proper balance for this sort of group is often more difficult to attain and sustain. Consistency is hard to achieve in this context, in my experience. Individual virtuosity becomes extremely important - the more original each individual "voice" the more interesting the result.