I have the same problem with allergies to anything but cotton. There's a lot of good info in this "I hate bras" thread. I have just ordered an "Unbra" without a latex band. It takes three weeks ot get one because they make them to order. There is a link in that thread.
Good luck!
kat