Perhaps we should think of Jesus and the fig tree that bore no fruit (see Matthew 21:18-22). There are varying interpretations of this piece of Scripture, but one of them might well be a remnant of the Gospels trying to preserve the "fully human" aspect of Jesus. We tend to overemphasize Jesus' divine aspects and to forget that He was also fully human and I find the songs to reflect that fully human aspect pretty well. Try to imagine a child with the powers of God and you will have some understanding to what Joseph and Mary went through and Joseph's fear of confronting Jesus that is evident in one of the versions of the song.

http://www.rapturealert.com/jesusfigtree.asp