Here's more info:

The song was adapted by Eric Von Schmidt, a blues-guitarist and singer-songwriter of the folk revival in the late 1950s. Von Schmidt was a well-known face in the east coast folk scene and was reasonably well-known across the United States. His chronicles of the Cambridge Folk era, are titled "Baby, Let Me Follow You Down", and describe the evolution of the song. Eric had first heard a song by Blind Boy Fuller called "Baby, Let Me Lay It On You." Eric von Schmidt credits Reverend Gary Davis for writing "three quarters" of this song.

First known recording by Walter Coleman in 1938 (Yazzoo 1991 B000008I25). It was changed to Baby, Let Me follow You Down around 1959 and became a feature in the coffee houses of Greenwich Village in the early 1960s. The song was sung by local heroes such as Dave Van Ronk. The song was later picked up by the young and up-and-coming folk singer Bob Dylan who (after signing a record deal with Columbia Records) made the song famous.

Another famous recording of the song was performed by The Animals in 1964, who changed the title to "Baby Let Me Take You Home".

An alternative history of the song is given on Arnold Ryens site "The Originals". He claims that the earliest version was by the "State Street Boys" (featuring Big Bill Broonzy). It was issued in 1935 under the title "Don't You Tear My Clothes." Washboard Sam recorded it under the same title in 1936.

Baby, Don't You Tear My Clothes

Baby, don't you tear my clothes
Baby, don't you tear my clothes
Push and shove me all night long
Baby, don't you tear my clothes

Baby, you can wear my shoes
Do anything you want to do
Push and shove me all night long
Baby, don't you tear my clothes

Baby, don't you tear my clothes
Baby, don't you tear my clothes
Wear anything I got on the table
Baby, don't you tear my clothes

Baby, you can wear my shoes
Do anything you want to do
You can push and pull me all night long
Darling, don't you tear my clothes

Hug and kiss me all night long
Baby, don't you do me wrong
Sit and listen to this lonesome song
But baby, don't you tear my clothes