"And many valley folk didn't like black people "because they do come from Cardiff" Let me try to explain:

As Britain never had a sizeable slave population, it was mostly in ports that coloured people began to congregate to form noticeable communities, initially of seafarers. Cardiff was a great seaport, exporting coal all over the world, and had one of the largest coloured populations in Britain. There was no love lost between the people of Cardiff and those of the coal mining Valleys. The joke above is that people from the valleys don't care a fig about your colour, but being a resident of Cardiff automatically makes you an "enemy".