"Your man" isn't an offensive way of referring to a third party. "I saw your man down the shops yesterday". It just means "the man we are talking about", and it's a very common expression, especially, but not exclusively, in Irish conversation.

Why should it be seen as insulting to insert "Rabbi" in the middle to clarify which "your man" is being referred to?