That line indeed appears in Mr Dylan's song, but he took it from a much older song (I am quoting from memory; I think I learned it off one of Richard Dyer-Bennett's records about 1952):

GO 'WAY FROM MY WINDOW

Go 'way from my window,
Go 'way from my door,
Go 'way, 'way, 'way from my bedside,
And bother me no more,
And bother me no more.

I'll give you back your presents,
I'll give you back your ring,
But I'll ne'er forget my own true love
As long as songbirds sing,....

I'll go tell all my brothers,
Tell all my sisters too,
That the reasons that my heart is broke
Is all because of you,....

Go 'way from my window,....

I have also heard Marlene Dietrich sing it. I don't know where it comes from.