I've just finished several weeks working on a project with a big chunk of Bellowhead. They work from scores at first, then allow their inner musician to take over and take the music where it's got to go to in performance. I was fortunate, being able to sight-read, my run-throughs took no time, making the project viable: if we'd wasted the same amount of time on me we did on those who couldn't read.
The dots are there to tell each other what we ought to start from, they're not some kind of Midi code dictating every nuance of a performance. That's where those who only work from a score go wrong, they're not musicians, they've traded their souls to a machine. It's like reading a book outloud in a monotone: you have to add your own interpretation and expression, to bring it to life.
So yes, use the dots, and perhaps have them available if your memory fails because the interpretation has strayed a bit far from the score. But in principal, know what you're doing and don't do it unless you do. You don't expect actors to read from their scripts on stage, the only difference is that they aren't constrained by the somewhat artificial constraint of the musical conventions. I know only too well that where we sing from in our brains is not where we speak from, but the principle remains the same: it has to be you performing, not the composer.