My own interpretation of the "queen of hearts" and the "not making hay" lines is that they both reflect the idea of summer, and its symbolism possibly links them. In this bright warm season (of youth?) she is industriously at work, while he by contrast is not - though the sun is shining and the chance may not come again. This idea of passing time that is gone forever also finds resonance in the first line of the poem, which I BELIEVE (though I'm relying solely on memory so please only one person at a time shoot me down if I'm wrong)is that it's not on an "autumn" day but an AUGUST day (check a poetry anthology - songbooks can get it wrong too). If so, it changes the opening season (though in August summer is nearing its end) and the song then progresses through time to late autumn/early winter, as do his hopes. And finally there is no season evoked at all, only ghosts. This could also be an oblique reference to the difference in their ages.

I actually once met a lady (now dead) who had known Hilda, and I'm annoyed now with my younger self for not plying her with questions! All I can remember of our one conversation on the subject (years ago) was that Hilda was a very charismatic character and many people besides the poet were attracted to her. The poem beautifully captures that elusive quality.