I always thought Martin was the " boss" speaking to his man(ie. servant).

Whilst this seems a reasonable interpretation DtG's posting above with it the other way round seems more likely to me, now that I read and think about it.

On reflection in another use of the word "Man', in Lou Reed's "Waiting for The Man", it's obvious who the "boss" in that relationship is.