Biographers believe the song (published 1855) refers to the hard times Foster was experiencing at the time. Both parents died in 1855, he was in debt and had to withdraw money from his publishers, his friend Charles Shirra had died in 1853, and for a time he was separated from his Jane.
Also hard times on the American labor scene.
Nothing to do with the potato famine or any events in the British Isles.