There's a lot of racism of one sort or another in society.
I rather suspect there were a fair few among the Remain vote who were to some extent racist. But they still saw enough reasons to vote to stay in the EU to decide t
And the same would apply the other way, people who weren't in any way racist who had other reasons to vote Brexit.