Casting my net...

This is killing me. Part of this song floated through my brain, and I can't remember it. The song talks about a club, a "terribly terribly private club" if I recall correctly. A "terribly terribly private club consisting only of me" or similar meaning words.

The song describes the membership, paraphrasing, we don't want "any of them. Or them. Or them, either." And when the song gets done with all the "thems" it is only the singer/author who is a member.

When I "hear" it, I hear a male voice, perhaps Joe Glaser, or a male group, like the Brothers Four or the Kingston Trio, or the Chad Mitchell Trio.

I did an Internet search on all of the words I listed in quotes here, but nothing like what I am looking for came up.

It's an ear worm, and I only have tiny pieces! HAAAALP! Thanks.

Saul