As a non-Englishman, I take the rhetorical "wasn't I?" as turning what would otherwise be a neutral answer into a smart-ass response, letting the questioner know the answerer is not only in the right, and being wrongly accused, but that the answerer is not impressed with the authority of the questioner.