I always figured "let her go" was spoken to himself, meaning, don't try to hang on to her mentally / emotionally.

Some absolutely stellar scholarship here. Makes the song that much more intriguing and wonderful.

And ditto on the Danny Barker version.

O..O
=o=