By all means, use your memory. I can't think of much to add to Don Firth's take on this. I always felt that the audience deserved to see a performer who appeared to truly care about the content of the songs he or she chose to sing. In a way, singing with feeling and respect for the content is also acting. I have seen singers - vocal groups seem more guilty of this - that do a very polished and professional act that looks and sounds exactly the same every single time. They have memorized the material, but to what end?

It takes a lot of preparation and practice to look at ease and comfortable on stage and with the performance. That preparation should also give you the freedom to improvise and vary your performance enough so that it doesn't become stale for you or the audience. Also, I personally like to see performers who don't appear to take themselves too seriously - who seem to really LIKE what they are doing and are happily sharing it with the audience.

Of course, at Don's age and mine, you start to become more and more grateful for retaining the ability to memorize anything longer than your driver's license number...