I put up with a similar situation for a couple of years (won't go into the details). I put up with it because initially I got an awful lot out of working with him, he was a great musician and very supportive in many other ways. But in the end, it gradually became more and more difficult, and less and less worth it, as I got more confident and maybe less dependent, and I left, for better things.

In the end, no one's irreplaceable.