In the UK, getting rid of them would be illegal although I think you can seal up after they have moved on. It could be quite different in the US of course but I'd still suggest checking up on this.

Another suggestion would be to find out if there are any bat conservation or wild life groups who could advise you or help you out.