No, you are not being self-righteous. In America, we call this getting up on our hind legs. Before you were frightened, now you are indignant. I think that's a good thing, but I hope if you actually have another bailiff on your doorstep, that you will remain polite but firm.

I used to work retail, and the company educated us that an angry customer is likely trying to put something over. (A customer who has made an honest mistake is apt to be flustered and apologetic, not angry.) The bailiffs may also believe that an angry person is likely a dishonest person.