Alright, as a first contribution re "small New England towns" -
When I first arrived in Boston for my initial moderately long stay there, the exisiting "modern" building code made it ILLEGAL to erect any building more than four stories in height. There were a few "historic" buildings that slightly exceeded the detail specification, but the taller ones were largely very light-weight wood framed construction.
The explanation was that the soil "bearing capacity" was incredibly poor.
Shortly after arrival, I found the historical marker (very nice bronze) purportedly marking the end of the pier where the events of the "Boston Tea Party" occured. It was about 5 city blocks (approx 1 km or 1/2 mile) from the actual shoreline ca. 1958. The "soil" between the original pier and the current waterline had been filled in by the dumping of the city's garbage into the bay for a couple of centuries.
The first really significant "waiver" for a "tall building" was granted for the Prudential Tower, which is - as I recall - 33 or 37 stories tall. To support this tower, reports were that it was necessary to drill through about 80 feet of mud (a.k.a. garbage) to reach bedrock, and then to drive piles an additional 80 feet or so into the rock. Some claim that it still remains "a little wobbly" but additional variances have been granted so there now are several "taller than four stories" buildings in the area.
In contrast, the university on the north side of the Charles river in Cambridge was built ca. 1860 or so with NO FOUNDATION. It's simply "floated on the mud." Although it has a fairly large "footprint" it's mostly only three stories above ground (with the central dome at approximately 7-stories at the peak(?)). A Master's Thesis by a student ca. 1950(?) determined by accurate(?) survey that this building has "tilted" so that the difference in elevation from one end to the other had changed, since construction, by 3 to 6 feet, and the entire building had "slid sideways" by a small amount, on the order of a few feet. It could be, according to those measures, expected to slide into the ocean in approximately 800-1200 years, however the odds on it catching up to the extensions of the "shoreline" due to the continued trash/garbage deposition (and natural silting from the Charles river) are practically nonexistent.
Although the design includes some "concessions to the soil" according to normal architectural standards, this building does apparently have reasonably good load bearing capacity. (see details at hacks.