While this isn't exactly a matter of time signature, a few of Conlon Nancarrow's player piano pieces put two voices against each other related by irrational ratios - instead of mere triplets, he has square-root-of-2-plets.

Study no. 33
Kyle Gann on Nancarrow
how to do human performances of Nancarrow

That isn't exactly a tune and it would take a lot of mandolins to play it.

More from the art music world...

Sing along with Brian Ferneyhough in "Bone Alphabet"
mindboggling score
finished performance (bloody hell, he's MEMORIZED it???)