There's a few more interesting facts about this song, that I came across in this piece on Bayard Rustin on the Early Music America website. (It's a very interesting article about Bayard Rustin, and specifically about his involvement with the New York City Early Music scene.)

The lyrics actually come from a book called 'Vedanta for the Modern Man' edited by Christopher Isherwood. Rustin would have come across this from his own interest in Vedanta, as a part of becoming acquainted with Gandhi and his non-violent protest philosophy. You can see the page in question from the 1952 edition here.

The poem is by Frederick Manchester (who was also involved with publishing other books on Vedanta). Its title is 'Sister Lalita'. Sister Lalita (aka Mrs. Carrie Wyckoff, lived 1859-1949) was a woman from southern California who became a Vedanta nun and donated her house for the use of the Vedanta movement. So what sounds like an 'olde timey' love song was actuially written by Frederick Machester following the death of a 90 year old woman whom he would have known from seeing her about the house when he visited as part of his own Vedanta interests. Note that Sister Lalita was known for her love of working in the backyard flower garden -- hence the 'flower among her flowers content' line. There is an article on Sister Lalita here.

Bayard Rustin opened side 2 of his original LP with this song, describing it as 'an Elizabethan song of unknown origin'. It seems almost certain that Rustin actually wrote the music (though perhaps only writing the melody and setting it over an actual old lute exercise, if the note from Seraffyn's LP mentioned above is accurate). In any case, the original words of the poem are:

I saw her as I came and went,
I saw her queenly, meek, and mild,
As innocent as any child,
A flower among her flowers content.

I come again, and in her place
Are silence and a vacant room,
And in my heart a sudden gloom
That I no more shall see her face.

There was a word I would have said,
Though what it was I hardly know;
I let the days glide by, and lo,
I now must say it to the dead.

(While researching things connected to this song, I also came across this New York Times obituary for Donald Mork/Seraffyn. Tragically, he died in a car crash in South Carolina in 1964, not long after the release of his one LP. The obituary mentions that he was wearing his medieval troubadour outfit when he died.)