Forms like a-bornin' were common usage in older "rustic" forms of English. Lots of traditional songs use them. Lines like "We'll go no more a-roving" and "Three maids a-milking" abound.

Even though Tom Paxton's song also has modern references like "Underneath my feet the subway rumbling," the old usage of a-bornin' gives it a nice traditional feel, no?