When I got married in a hotel the other year, the registry office people were quite strict on the principle that there were to be no religious overtones to the event, no hymns sung etc.
So, as we were signing the register, the congregation listened to the following over the speakers;
Sterephonics & Jools Holland - First Time Ever I Saw Your Face.
I couldn't think of better songs at the time (although there were many on the shortlist...) that could suit the wedding.
I suppose atheist hymns are an oxymoron, although I don't like putting atheist on forms when asked, (I usually put other and then say not applicable.) For me, atheism is portrayed by many as a belief system in itself. The belief that there is nothing. Einstein had a problem with that because he said that atheism ultimately is the belief in chaos and as we experience the laws of physics working, then whatever the answer is, it certainly isn't chaos.
If people use the term atheist and then put labels to it, then I would have to say the term no longer describes me and irreligious is a better term. (I joined the church of the flying spaghetti monster the other year, but so did a few colleagues and our childish aim was to make the government body we work for state it in the staff statistics page in the annual report. We pastafarians get about you know!