In his biography of James, I'd Rather be the Devil, Skip James and the Blues, Stephen Calt recounts asking James about the lyrics to this song:

"What's If You Haven't Any Hay about?" I asked.
"Out in the country we used to go hack-ridin'; hay-ridin'. Sometimes there'd be someone who didn't have any hay. So he'd best get on down the road." He chuckled, and began citing a low-down lyric: "You heard Make Me a Pallet on the Floor? Well, I styled it this way: 'Give me the keys to your back door. So I won't be bothered 'round the front no more.'"

(Calt, 1994: 265)