I think the posts above have covered most of this.

From the mid 70s onwards we hit a negative spiral with people dropping out of the scene as they started families or took more demanding jobs and new people coming in in far smaller numbers. With club income falling singers nights which formerly happened once or twice a year soon became one or twice a month and then it was the guest nights that were once a month.

The pressure to keep finding and learning new material while trying to revive a slowly shrinking club going really took the fun out of it.

Now we have a small core of fairly strong clubs booking high profile guests but a lot of clubs that primarily have singers nights with few or no guests and, with some honourable exceptions, are pretty much as described by the OP.