My assumption is the "hard times" was referring to the constant and recurring cycle of boom and bust which the 19th Century was particularly vulnerable to, and to the fact that most times, the poorest people didn't ever get much from the boom times, but went bust fastest when the balloon burst. History books are replete with references to "the panic of 18xx." Those panics were the result of a lot of factors, of course, but unrestricted capitalism and the prevalence of speculators, adventurers and a good number of frauds were a big part of the cause. For a different folk song connection to these mercantile matters, look up the history of "Oleanna." That project was an attempt to set up a colony -- or at least settlement -- in Pennsylvania (I think), where the land being purchased by Ole Bull, champion fiddler and would-be benefactor, was in fact not owned by the sellers, who ripped Ole off and left the settlement high and dry. At least, that's the way I heerd it!