Well, now, I wouldn't want him to be diddling with his wife in the Oval Office, either. What he does when — and where — he's not supposed to be actively conducting the business of the Presidency is his own affair (so to speak). Same as any other salaried employee.

On the other hand, the argument can be (and has been) made that the President's 24/7 conduct is a reflection on the nation as a whole... and we've certainly had more than one as-whole in office.