I always heard it as:

Bring no ill will to him nor us, my helpless babe and me...

I often sang this song to my little girl when she was wee... but I always changed the last verse deliberately to: A little while, and then the world is full of work to do (as oppsed to "a little rest"). And by that, I meant the growing up years will be so short. Soon this litte child will be grown up and working, and worrying about the worlds troubles.... only a little while.... enjoy that little while...