The same controversies were aroused by the pinching of the Elgin Marbles from Greece, and countless mummies and other tomb artefacts nicked from Egypt. I'm afraid colonials thought the world was theirs to loot and grab at will. I wonder if these things would be well cared-for if returned, and available for research and study? Whether or not, you can't justify hanging on to stuff you took without permission, so in general I'd be all for handing them back, and offering appropriate apologies to the original owners.