Thanks to Google Books, the first publication of Lowry's hymn is now online. Bright Jewels for the Sunday School

I also found an earlier printing of the poem, from England!
The Christian pioneer, v.20, 1866. p. 39.

It is also in the New York Observer, Aug. 27, 1868. Title is "Always Rejoicing" Above the title it says "For the New York Observer" at the end it says "Pauline T."