If the "to" is sous-entendu, or whatever the English phrase is for goes without saying, it is still part of the infinitive. But I should look that up.