I think you may be stuck with them for the rest of the season. Only the queen overwinters so you should try to find her as she emerges in late winter or early Spring. She will probably abandon the old nest, anyway, and find a new nesting site.

Right now the workers are especially active. The colony has probably grown to include a large number of yellow jackets which require vast amounts of protein. Stay away from them!

I've seen some great traps made from large pop bottles. These work quite well. In the meantime, don't bug them.