I semi-know tons more songs than I could possibly commit fully to memory, and of the hundreds I've learned by heart, if I don't keep actively singing them, I space on large chunks. I see nothing wrong with using a book--it's about singing the song, not testing your memory skills. As others have said (quoting Martin Carthy?), the only way to dishonor a song is not to sing it. (I think there are plenty of other ways, but you get my point.) As long as you are otherwise prepared to do the song justice, have at it! Those who would snub their noses have their priorities screwed up. And probably keep singing the same songs ad nauseum.

On the other hand, "poor memory" often means that one just hasn't learned a good approach for memorizing songs, or hasn't applied it systematically. I'll bet there are lots of songs you do know by heart, putting the lie to "poor memory". (Which is not to say that because there are tried-and-true methods for memorizing, you should have to memorize every song before you perform it informally--that's just hogwash.)

There are multiple threads here giving tips on how to memorize lyrics, so we needn't digress into that here, though some kinds souls might refresh those threads, or link them here.