In my New England home, the largest town building was the Congregational god-box, or possibly the Episcopalian one. The grade school was a close second. With the expansion of the 50's and 60's, the Town Hall grew larger, and the high school as well, so they were both probably larger in square footage, but much lower-built, than the churches.
We had a few top-of-the-hill mansions that were bigger than the churches, though, sort of temples to Mammon I suppose... ;>)