It may be that your rhythym is quite fine solo, but not necessarily easy to for someone else to follow, due to phrasing, mixing around upstrokes and downstrokes, etc. (this seems to be my situation, anyway). It might be helpful to stick to a very regular, almost drum-like meter on your instrument (guitar, I presume?) during your partner's breaks. It also helps if you both know the songs, by which I mean every note and every pause, with certainty in your heads. Sometimes minor variations in phrasing that folks may not know exist can bump against each other when collaborating. Keeps life from being boring.