You make some great points. Back in the 80's we used to hold annual concerts (more like a mini indoor festival) as a fund raisfer for the radio station. All the artists appeared gratis, but of course we were giving them air play. The problems we ran into were just as you described. Programs, publicity, sound crew, food, etc. were not cheap. We were lucky to have a "free" performance space, but it was in a building on the campus of the university that did not normally hold outside events. We had a tough time drawing people to the event. At the end of the day we would raise a couple of hundred dollars which would have been a loss if we paying the artists.
A few "benefits" to consider is how much good will and other opportunities will the event create? While we did not raise a lot of money, we did raise the consciousness of the radio station in the community and developed ties with the musicians that continue to this day. Often a benefit concert will be an introduction to an audience about your specific cause. Harry Chapin did a wonderful job of this when he used to talk about the hunger issue during his shows.