"If they are that way inclined, don't they have a citizen's duty to stay away. "

That's a bit unclear what you mean - if you are referring to the 'naughty' people - well the whole thrust (sorry!) of the argument is that they are 'unable to control themselves in the way that a normal adult is expected to'...