To answer you question; I would have no problem with it what-so-ever provided the weekend occupants respected the rest of us in the neighborhood and did nothing to disrupt our enjoyment of our homes and neighborhood. I, as you, would make no distinction as to ethnicity, religion, or any other categorization we use today. Truth be told I would welcome diversity as I always learn something new from those different than I.
In fairness your comment "Only the zoning board cares because their little beauracratic feathers have been ruffled" seems a bit short sighted. The concern may very well be that the establishing of the precedent by way of the variance may create real fears in the community at large as to what other types of transient housing might follow and therefore the zoning board may only be reflecting the community's thinking. It may also be that the collective vision of the majority of the community at large may be for a different type of growth in this particular area. Just because this is contrary to the vision of some doesn't make it discriminatory or retaliatory.