"Even if I believe it's righteous to stone adulterers, and I do, doesn't give me the license to go over and stone my neighbor lady. God hasn't given me the jurisdiction to do that."

As I understand it, it was normally adultRESSES who were stoned to death (and are, I believe, still stoned to death in some parts of the world today). That was because their societies were based on patrilineal descent. Men passed their 'property' on to their sons (i.e. male, genetic offspring) after death so they needed to ensure that any male heirs were descended from themselves and they didn't leave all of their worldly goods to some 'bastard' resulting from the wife's indiscretion. So adultery was not originally a crime 'against God' but an economic one based on male dominance.