I always heard "conspire" as a witty hyperbole on the part of the speaker: it's just we two with our secrets on the one hand, and the cold world outside on the other--deliberately underscoring the fireside coziness and the light-hearted semi-nonsense throughout the song. (Parson Brown can "do the job when he's in town?"But he IS in town: He's right there, you just built him.)   Of course when faced with an apparent anomaly, listeners try to make it fit, but I confess it never struck me as the least inappropriate.