I don't think you're 'with' him if your contradicting him. I can't say for sure why they had a cow over it except they don't like him because he's not one of them, whatever 'them' means.