Fay, as it is spelled in older versions, has many meanings, but a common one was faith, or faithful.
Fay itself also was used for a fairy. Was the author just re-inforcing the line?
Fey commonly means a little 'touched'- odd, crazy, doomed, etc.

On the other hand, it could have been used by the writer as a meaningless word to fill out his rhyme with day.
Fairy was used as a term of endearment in the 19th c. Fairybabe also a term of endearment (was the original "Fare thee well, my fairybabe?). See OED for these meanings and variations.

To interpret the meaning of words in an old song in this manner is, of course ridiculous.

Ofay, origin unknown, possibly pig Latin for foe, is a modern (since WW1) word, hence does not apply to this song.
Lighter, J. E., 1997, Historical Dictionary of American Slang, vol. 2.