I'm surprised no one has brought up the similar concept behind "offhand."
The logic of "offhand" and "out of hand" both seem to spring from the premise that if you have something "in hand," or "at hand" (as contrasted to in your house, or even in your saddlebag), it can be instantly dealt with. That goes for an opinion or an espousal.