Well, just look at the lyrics: "Run all about, and say he make his name". Emphasis on "say he make". It's not the Duke who is claiming that Invader is promiscuous, it's Lord Invader himself who is claiming that he is promiscuous. Some of it may be exaggeration, but I've got a feeling that most of it is true.