Hi, Matty - I think I'd start with a song circle or jam session, whichever is more interesting to you. If you have four people who come regularly, you've got enough for a start. Just keep inviting more people until you've got a fair-sized group. Start small, and keep building. After a while, you may have a group big enough that you can invite performers for house concerts. Bakersfield is big enough that I think you should be able to do it.
Good luck.
-Joe Offer, your neighbor in Sacramento-