Agree with Diane about reading and learning music by ear both being useful assets. Came up the "Always must have music in front of me" way, but then started going to classes where the favoured method is learning by ear, and I'm slowly getting better at it, and at improvising (eg when you get that nod from a singer to put in an instrumental break in their song).
As for music sounding "rough", think of Eiza Carthy's "Rough Music" CD: sounds like they're having an amazing "jam" together, but no hint of instruments or voices being out of tune or sloppy playing. Hard to define, but you know what it is when you hear it.