"I, as you, would make no distinction as to ethnicity, religion, or any other categorization we use today. Truth be told I would welcome diversity as I always learn something new from those different than I."

That is very true, but you are looking at one side.   Diversity is not the issue. I moved to the town I live in because of the diverse makeup of the community.   However, I would have a legitimate concern about allowing a house that would have such an arrangement because of the "transient" nature of who is coming and going. Home security is a big concern in this day and age, and while I am sure the occupants in this situation would be above suspicion, changing the variance would have implications as Ed pointed out above - and would I want to share a neighborhood with houses that have different occupants that I don't know coming in each week? It is one thing to say they would be respectful of the neighborhood, but what guarantee can be made?   If my neighbor causes issues in my neighborhood, I know who to go to and how to deal with it. Can I say the same thing about a "transient" residence?

With that said, I think there are situations where it works.   Each summer I go to Ocean Grove, NJ which has lovely bed & breakfast and "summer renters" mixed in with year round residents. You can't get more "transient" than that! The town also has a significant religous community, but that is not the sole reason people go to the town. The difference of course is that new home owners know the situation when they move in.   In this case, you are trying to overturn an established neighborhood.