Not to worry Keith - it specifically indicates he's a young man at the time of the song.:

"In city or country, a girl is a jewel
And well made for holding, the most of the while
But any young fellow is really a fool
If he tries at the first time to go a bit far


You could always amend the last verse so it's got you looking back at a happy memory. For example:

Well I chanced for to go the town of Dungannon
After years that I wandered this world all around,
But I sat on my own there in fond reminiscence,
For another like Annie I never have found