I agree that talking it over with other singers in the group would be useful. In a song circle I go to we lost several people because of one rude man who wouldn't keep quiet. I lost my temper one day at him actually and stopped my song halfway through to tell him to be quiet. I'm still embarassed and annoyed with myself for doing it and wouldn't recommend ot to others as a solution. I kept going to the circle because I knew that others were irritated by him so I wasn't struggling alone and therefore I didn't have to take it personally.

This man that you are having trouble with does at least wish to talk about the problem which is a good thing and privately he may be more amenable to sweet reason than publicly. If he continues to bother you afterwards do your best to ignore him because he's being rude and seek support from others in the group. Another person asking him politely to lower his voice because they want to hear you sing might do more.

Another quick thought that occurs to me is that he may be losing some of his hearing and therefore be unable to judge volume. Are there other occasions when his volume seems inappropriate for the circumstances? Whatever happens keep on singing and singing and singing.