I reckon you're onthe right lines, Neo - fidddlers tend to be the best musicians around in the folk scene, and they can't see what they are doing too well mkst ofbthentime - and they don't have frets.
At times in a session, playing a guitar, I'll be looking at my left hand, but that's to check what key I'm playing in, and what chords I'm playing. If I tried working out where my fingers were supposed to go while I was playing, I'd go crazy.
While you're working out something you look at the finger and the finger boards, because you want to know where it was you found that note you needed - but once you're playing you let your hands sort it out on their own. And, if you're not on your own you look at the other people you're playing with.