The phrase is derived from an old widely disseminated planting rhyme.

I've seen a version in George Ewart Evans' book Ask the Men who Cut the Hay (where it's said of 'corn', ie wheat to Americans).

There's a version on an American seed catalog website where it's called a corn-planting (ie maize-planting to UK-ians) rhyme:

"One for the cutworm, one for the crow, one to rot, and one to grow"

...and no doubt many others. The hard life of knowing you couldn't eat, or use as feed, the seeds what you planted, but that pests would.