Personally, I don't have a problem with the Fol Rol Riddle refrains mainly because the source singers I learned them from used them, but I agree that you should not sing them if they make you feel awkward but you might otherwise really like the song and want to sing it.
One solution, and I seem to remember that you are a guitarist, Matt, would be to play the refrain as an instrumental.
After all, that is what Shirley Collins & Davey Graham are doing here.