Up until the 1830s all paper was made from rag. Rag pickers were common, and it was considered a very low end sort of job. With the industrial revolution and the development of the Fourdrinier paper making machines (putting out continuous rolls instead of hand-made sheets) the rag pickers couldn't keep up with demand so others types of paper were developed and wood cellulose won out. Linguistically things stick around a lot longer than the thing that they describe, and I think the reference to "rag" or "rag tag" has its origins in that kind of work.

SRS