OK, I'll get serious about this for a second.
I don't think the argument is illogical, but I am not necessarily defending it either. This is what I believe underlies the fear.

In Guantanamo, these individuals were isolated. They were isolated from any general prison population, such as exist in the US mainland. No prisoners convicted of armed robbery, murder, embezzlement or drug dealing were being held with them, such as would be the case at, say, a Leavenworth. They were isolated from outside contact with civilian population, which would be much more difficult in a place like Leavenworth with its teeming population of inmates and the steady chain of visitations, letters, phone calls, etc.
And this gets to the heart of it. I think many consider Islamic Fundamentalism to be a form of contagious insanity, and that these people are considered infected carriers. The idea of introducing them into a population of individuals already predisposed to violent anti-social activities and having them become infected is one that gives many pause.