Silly me. I thought it was simple.

We give money to the EU because we are rich and they, or part of they, are poor. But there is a significant part of the UK that is also poor. When we give our money to the EU we deprive the poorer parts of our society of that money. Of course at the same time there are parts of the EU that are very rich.

No surprise that the rich parts of the EU do not contribute to the poor parts of the UK.