Just looking at a few variants-
From the version "Cold Blows the Wind," Burne 1886.

O don't you remember the garden-grove
Where we was used to walk?
Pluck the finest flower of them all,
'Twill wither to a stalk.

"How Cold the Winds do Blow" Broadwood 1902

It's down in yonder garden, love,
Where we were used to walk,
There's finest flowers that ever grew
All withered to the stalk.

"The Unquiet Grave," Greenleaf and Mansfield, 1933

Down yonder meadow where the grass grows green,
Where you and I used to walk,
The prettiest flowers that ever we had seen
It is withered unto the stalk.

D in Child, Buchan's MSS 1

Mind ye not the day, Willie,
Sin you and I did walk?
The first flower that we did pull
Was withered on the stalk.

H Baring Gould MSS, XXXIII

O don't you mind the garden, love,
Where you and I did walk?
The fairest flower that blossomed there
Is withered on the stalk.

"The Unquiet Grave" Peacock

You go down in some yonder green grove
Where true loves used to walk,
And there you'll find so fine a flower
All withered unto the stalk.

etc.

Reminds me of a verse by Francis Thompson-

The fairest things have fleetest end,
Their scent survives their close;
But the rose's scent is bitterness
To him that loved the rose.
(Daisy, 1893)