I think it's religious sects' insistence on calling their polygamous living arrangements "plural marriage" that many people have a problem with. There are plenty of people who practice some sort of multi-partnered alternative domesticity without calling it anything at all. Nobody cares what goes on in their bedrooms as long as it doesn't frighten the horses.