The major early 20th century song collectors paid their informants for their time, in cash or in kind (not infrequently including alcohol and a nice dinner) or, indeed, in both. As a rule, this was a private matter between collector and singer, but surviving correspondence can sometimes provide details. Of course, there were a lot of people who just picked up the odd song here and there; whatever arrangement they had will have depended on their relationship with the singer.

They weren't 'buying' the songs, but paying the singers for their time and trouble. In cases where the songs were published, this was typically done at a loss. Published versions were frequently edited, collated and arranged for accompaniment, and the editor was entitled to copyright in the resulting version. In practice this was rarely insisted upon, though normal courtesy demanded that permission be obtained for commercial use. On one occasion, Sharp declined permission (he wasn't happy about the proposed use) and was roundly criticized by the gentleman concerned for asserting proprietory rights over a traditional song. Sharp pointed out that only that particular form was at issue; all the fellow had to do was go out and spend his own time and money collecting one of his own; from the same singer if he liked.

Towards the end of his life, Sharp actually started to make a modest profit from publishing, but he was very unusual in this. George Gardiner not only paid his singers, but the collaborators who took down the tunes for him; and the only 'commercial' selection he published had piano arrangements commissioned from the young Gustav Holst. No cash profit there.

That was a century ago. I don't know what precise arrangements might pertain nowadays, but I can tell you that folk music books rarely achieve large sales and make only modest profits; and that small publishers typically pay royalties as a percentage of net, not gross, receipts. If you edit a book for somebody, then (if you haven't actually done it for nothing) you will either get a fee at the time or a royalty on sales; not both.

That, of course, tells you nothing about what arrangements might exist for payments to living source singers/musicians or the heirs of deceased ones in the case of material provided by them and included in new collections. It is necessary to obtain formal permission for the use of material, but I don't think there's any hard and fast rule beyond that. Each case is likely to be different. Note that I'm talking only about print here; different rules apply in sound recordings.