Did I challenge his statement in any way that was wrong?
It is not wrong to challenge anything, but anyone reading your response to his statement would be quite clear that you were challenging it, and your denial just taken as more evidence of your dishonesty.