Here's something interesting I discovered. I think it ties in with what you are saying at least to some extent. And I think this is a brain wiring kind of thing.

My first instrument was the piano. I was not a particularly gifted or enthusiastic piano student, and my instruction was group lessons given at my elementary school once a week when I was in the fifth grade. I didn't like to practice, and I never got very far, but I did pick up some of the rudiments of music theory.

I never was able to read the music very well, because I'd pick through it just enough to memorize the piece (and I'd memorize some of it when it was being played in class), and once I had it memorized, I only pretended to use the sheet music. I gave up with the piano after about a year, although a few years later I tried to start with private lessons, but the teacher started me out playing Bartok. That didn't work so well, I guess because Bartok was really hard for me to understand or memorize.

For many years my instrument was the recorder. I was never able to read music for the recorder. I did try, but my efforts only resulted in frustration. For years I played early music (renaissance and early baroque), and learned by listening. I was only able to learn the treble line, and I was never able to play some of the early baroque pieces that I loved by composers like Teleman and Vivaldi. Eventually, I started playing traditional folk music almost exclusively, which I learned by ear.

I had pretty much given up on learning to read music. But to my great surprise, when I started learning to play the accordion, I discovered that I could read music for that instrument. Not sight reading, but enough to be able to learn new pieces, and to use the music as a reminder if I forget how part of a piece is supposed to go.

I think maybe the brain wiring aspect of this is that with the recorder, several fingers are used to make one note. And there are many combinations of finger configurations required for even the simplest of pieces. With the accordion, one finger is used to play one note on the treble side, and one finger is used to play one chord on the bass side. My brain can translate the note I see on the page into an understanding of which one finger to use, but apparently not for several fingers at once.

Also, I think reading for the piano was a problem because there were too many notes for me to really see and understand at once. I have Attention Deficit Disorder, and apparently this effects my ability to handle a lot of sensory stimuli. With the music I use for the accordion, there is only the melody line, with letters for the chords above the dots.

So in my case, maybe the problem needed to be overcome by switching to the right instrument. And I must say, while practice felt like work on the piano and on the recorder, on the accordion, it all just feels like playing music. I usually play for at least a couple of hours almost every day, and it never feels like work. I love every minute of it.