I always think of the Irish Potato Famine when I sing it or hear it, in the first place. Cabins and all.
Given the date it was published, that could well have been in Stephen Foster's mind too. There were enough refugees from the famine around. But it might have as well been some American hard times, and probably was. And it's as topical now as ever. Here's a story from today's Guardian, for example.