Even if I was allowed to own or not own slaves in 1860, I would have fought against your right to own them. It wouldn't have been enough to just refuse to own them myself. The "live and let live" attitude you suggest doesn't bode well for the slaves of 1860.