I know of no evidence that the number actually had ever been much greater than it was then, or is now.

'Source singers' as we know them now - people who know dozens of songs by heart - maybe not. But there very definitely were more singers, because that was the only way the songs could live - by being sung. A single can sell tens of thousands of copies without anyone singing the song. If a broadside ballad sells tens of thousands of copies (and some of them did), tens of thousands of people were singing that song.