What makes the difference between people who can pull off this sort of thing and those who can't?

In my own opinion, part of the answer lies in two areas; the craft of lyric writing, and the skill of the composer of the melody. The lyric should be personal enough to be believable, yet general enough so that the audience can say "I've been through that." or "I can imagine what that's like." At the same time, the story sould be compelling, or told in a compelling fashion. It should have the audience wanting to know what happenes in the next verse, or how the story ends, or the Eternal Question the song presents.

The melody needs to be as interesting as the lyric, too. It's no good to have a great lyric and a melody that goes nowhere, or is bland. Ideally, I should think that the songwriter wants to find a melody that everyone will leave the concert humming.

Just my musings.
Jim