Properly applied, and used within the temperature and chemical exposure limits that are fairly obvious for a material of its kind, the JB Weld can be considered about as permanent as any other sort of patch. Assuming a sheetmetal pan, the JB is strong enough to hold together if the metal flexes within reason, and if the metal that's left around it ain't all gone there's no good reason the plug of JB shouldn't stay in place.

Proper application does require fairly good cleaning of the surfaces you want it to stick to. A good methanol or lacquer thinner wipe is one of the recommended methods of cleaning.

If the hole is a straight through puncture, just plugging it with something that will stop the oil from running out is all that's needed. If the hole is a crack or tear that has "sharp ends" where the crack could grow, of course an old hand like Bobert will have drilled "crack stops" at the ends of the cracks. A proper crack-stop should have a diameter at least 3x the metal thickness as a rule of thumb. For sheetmetal of the thickness I'd imagine for an oil pan, about a 3/16 or even 1/4 inch diameter drilled hole that cuts off the end of the crack, or that's placed so that any growth in the crack obviously will be "into the hole" should be adequate. Of course then you'll have to plug the crack-stop holes too, but the JB is just a cork to keep the juice on the inside.

John