You did right. Since he kept coming it seems to me that he wasn't offended. We all screw up and most of the time we are not aware of it. A friend will gently urge you to change your ways.