Found it!

The Brookside

by Lord Houghton

I wandered by the brookside,
I wandered by the mill;
I could not hear the brook flow,-
The noisy wheel was still;
There was no burr of grasshopper,
No chirp of any bird,
But the beating of my own heart
Was all the sound I heard.
                                        
I sat beneath the elm tree;
I watched the long, long shade,
And, as it grew still longer,
I did not feel afraid;
For I listened for a footfall,
I listened for a word,-
But the beating of my own heart
Was all the sound I heard.
                                        
He came not,-no, he came not,-
The night came on alone,-
The little stars sat one by one,
Each on his golden throne;
The evening wind passed by my cheek,
The leaves above were stirred,
But the beating of my own heart
Was all the sound I heard.
                                        
Fast, silent tears were flowing,
When something stood behind;
A hand was on my shoulder,-
I knew its touch was kind;
It drew me nearer,-nearer,-
We did not speak one word,
For the beating of our own hearts
Was all the sound we heard.
                  

Note "He came not......", another reference to this poem suggests it was a sonnet written to Charles Lamb.

Now I read it again, the words here do scan to the tune, so I'm not sure why I changed them!