Oh, and I meant "watch with your eyes," not "with your ears" (alone).   If you really want to match the phrasing and timing of a singer, that's better than just listening.

But I think you're right about why some people develop idiosyncratic timing. Some are used to singing a cappella with no rhythm accomaniment, and that can work well for some kinds of 'lyrical' song stylings, but it doesn't tend to work well in jam sessions.

BTW, I like the idea of folks maybe switching instruments or picking up unfamiliar ones for Bobert's Bad Band!   I could try my hand at the violin again, many decades after I quit lessons because I sucked so badly at it!