They make strips of prickly stuff to put around the top/inside of chimneys to discourage them from moving in. I looked the topic up a few years ago when I had a bunch in my chimney, but I can't remember any suggestions for chasing them away.

I did learn that if they start 'whomping' around, it's not a good idea to turn on the light (damper open) because the dang things seemed to think they needed to fly toward the light and I spent a long time catching them and throwing them back outside where they belong.