Michael.......I do know exactly what you mean and it is made worse by the huge amount of "novice" info that is passed out on every broadcast. The justification is always "the need for any new viewers to understand" but the result tends to piss off long term fans. When I watch a sport which is unfamiliar I would prefer to learn as I go by figuring it out for myself. Maybe I'm weird.....but I already know that.........but I think you get into an activity deeper, if also slower, when you have to spend some time at it.
I drive Karen crazy by making comments about what is going on before the commentators do. On some sports, I watch with the sound off because I can't stand the drivel. Some announcers drive me nuts anyway which is why I can often be found watching the event on TV and listening on the radio. This is especially true with auto racing.
Sadly Michael, you are stuck with it in most cases so I suggest you start talking to your TV. I do that and it has the added benefit of making your friends absolutely sure you have gone over the edge.