This actually is an issue in which the dividing line doesn't neatly align with "whether we vote left or right".

Actually, when you actually apply logic in regard to matters of political and social controversy, that's by no means uncommon. Whole ranges of issues get bundled together which have virtually no logical connection, and we behave as if being agreed, or the reverse, on some of them must imply lining up the same way for all or most of the others. And being herd animals we tend to actually behave in that way.