Then there is the discussion of difference between hymns and modern "praise choruses". I would argue a good hymn has a more complex structure and at least two verses, in contrast to the repetition of the "choruses". It is hard to project a good hymn up on a screen and include everyone in the singing, and the music requires a modicum of skill (e.g. being able to follow the melody notes) to sing on first exposure. A good hymn, IMHO, requires that you bring your brain to church, thus the theology is important; otherwise the touchy feely happy happy joy joy doesn't cover much of reality for most of us.
In the Presbyterian tradition the "classic" hymns we know now were once rather controversial, because the Calvinists insisted that only the Scripture was appropriate for singing in worship, not the "words of man" or pretty poetry ("How Great Thou Art" would have been anathema 300 years ago). Thus their earliest songbooks were Psalters, settings of the Psalms either 'canted' or laid out to rhyme in English.
The Gospel music tradition springs from the revivals in the 19th century where a more spontaneous, emotional tone was called for. They are more individualistic as well - most Gospel hymns are marked by "I" "Me" and "Mine" language (cf. In The Garden, Amazing Grace, Just As I Am, and thousands of others). The older tradition - and some newer ones (which I like) are first and foremost about God or Jesus, and only secondarily about "us".
For me the dirge-y hymns often get to me the most - "O Come O Come Emmanuel" and "Ah Holy Jesus" can reduce me to tears. I think that Isaac Watts was perhaps the greatest songwriter/musician in history, but he is mostly overlooked or forgotten. "When I Survey the Wondrous Cross" is an absolute gem. And "O God, Our Help in Ages Past"
Among more contemporary hymns, someone has already mentioned "God of Grace and God of Glory" which although written in 1930 sounds like today's newspaper to me (lines like "Cure Thy children's warring madness, bend our pride to Thy control; Shame our wanton, selfish gladness, rich in things but poor in soul; grant us wisdom, grant us courage, for the living of these days" get me every time). Brian Wren has written some beautiful hymns also - a line like calling God "gracious donor of our days" has volumes of theology in 5 words. Much global hymnody that has come to the attention of American churches in recent decades is powerful and graceful as well.

Finally, I would add for you folkies that I think the church is one of the few places in American society where people gather and sing on a weekly basis, regardless of ability or age, not for performance or remuneration but just to satisfy the spiritual necessity of worship. It is, among many other things, a unique community of music.

Peace,
Bo