Very few people have accurate pitching all the way through a song. It's pretty normal for unaccomperated singers to 'drift'.
I'm with Breezy and others - most don't notice, and anyway who cares?
Just sing!