"Is there a square foot of audience size to number of speakers equation?"

The issue is less about number of speakers than it is kind of speakers. For most clubs, pubs, etc., two speakers will generally be fine. Reasons you might move to more speakers would be a room with a significant rake or a very wide room. With a wide room or playing outdoors, you would typically create a simple left and right cluster made up of two or more speakers each. This way you avoid the need for calculating delay settings.

A pair of well-constructed 12" wedges will suit the needs of most bands, like maybe the Ramsa A500's.

Monitor mixing is tough, especially if you only have one mix for a whole band, and every musician wants to hear something different. Investing in the ability to have at least two separate monitor mixes is good if you can swing in. There are so many in and outs... and this is why the monitor mixer is always the first one fired off a tour :) With monitors, less is better. Lower volume saves the ears and prevents territorial fights over the mix. Generally speaking, go with whatever works for the band, whether it seems wacky or not.

But often: Lead vocals should be on top, with whatever instrument is most crucial to keeping the band in tune and on track just behind the vocals. (Rhythm guitar is a big contender.)

An acoustic group used to balancing themselves without reinforcement needs to be able to hear everyone (but inevitably there's a monitor whore who wants to drown out everyone else.)

A shy vocalist may need a little sweetening of the house effects in their monitor to encourage them, but generally it's wise to leave effects out of the monitor mix. A quiet vocalist will usually project better if you bring their level down in the mix a bit -- as long as it isn't going to mess up everyone else on stage.

A fretless instrument may need more of themselves in the mix, but sometimes it's simpler to make sure the banjo player is standing to the fiddler's left.

Enough rambling...