I suppose that depends on which version you're speaking of. The one I'm most familiar with goes:

[previous verse:]
...You are a false, deceitful man, I notice plain,
For it is you that has caused my poor heart to wander
And in your comfort lies no refrain.

So I'll go down to some lonesome valley
Where no man on earth shall there me find,
Where the pretty little small birds do change their voices
And every moment blows blusterous wind.

So come all young men that go a-sailing...


Granted, in this version, the penultimate verse is unclear about who is speaking and what the speaker means exactly. But I don't see how it could be taken as "rallying", regardless of who's speaking. For the final verse to imply a later rallying would be inconsistent with how this floater functions in other songs: to summarize the preceding example in a generalized form, like a moral. The positive variant just doesn't fit.

More evidence that it's a floater is that "sailing" or sailors are nowhere referred to in the previous verses. He might not have recognized her because he'd been away some years at sea, but we have no cause before this point to assume that. The banks of primroses imply an inland stream, not a seaport.

Side note: the lines about the birds and the wind seem rather conflicting, in both tone and metaphor. We'll ignore that pretty little small birds would have a tough time singing when every moment the wind is blustering; they're probably busy just hanging onto the perch, and their feathers. Though the wind would at least explain the change in their voices, should they attempt it.

For what it's worth, I believe it is the woman speaking, given that she is "I" in the previous verse, the mode of expression is a bit florid for a man, and she says "where no man on earth", while "where no one" would be more natural for a man. He also doesn't come off as one who would have difficulty rebounding.