I am skeptical that his career was that great. He urged the surge in Iraq and was a hawk beyond reason. I think that his affair should be held against him since I believe that public officials who do that can't be trusted.

This is redolent of John Edwards in that the ego becomes large enough to cause a downfall and a small character emerges.

I had reservations about Bill Clinton too. Even Eisenhower and Roosevelt had affairs.
Eleanor was hurt badly by the Warm Springs affair with Lucy by FDR. He died in another woman's arms.

Kennedy was known as "Jack the Zipper".

You can separate actions from the person but still I think this stuff matters regardless of how important that person becomes.