It now seems to me that the word we sang was "prettiest," not "finest."

In any case, the words of a Virginia version - which seem to be rather older than 1907- appeared in the Richmond Times Dispatch Sunday Magazine (June 14, 1936), p. 5, as sung by Martha Davis of Harrisonburg:

There was a tree on yonder hill,
And a cu-ri-ous tree it was, sir.
And on that tree there was a limb,
Limb on the tree, tree on the hill,
And a cu-ri-ous tree.

And on that tree there was a limb....

The final stanza is,

And on that bird there was a feather,
And a curious feather it was, sir.
Feather on the bird, bird in the egg, egg in the nest, nest on the bough,
      bough on the limb, limb on the tree, tree on the hill,
And a cu-ri-ous tree.