I've only just seen this thread, so apologies for belated and for repeating anything already said. However, what you describe is fairly common among unaccompanied singers. I do it, the late Joe Kerins did it and the wonderful Elizabeth Cronin did it. And if you can sing as well as she did, you're doing alright.

Unless the effect is very noticeable to your audience I seriously wouldn't worry.