"To call a spade a spade" not only predates slavery in North America by quite a bit but harks all the way back to the Ancient Greeks and doesn't have racist origins, occurring in the work of, among others, the playwright Aristophanes, and is still commonly heard in modern Greek. The original phrase seems to have been "to call a fig a fig; to call a kneading trough a kneading trough," applied to someone who spoke exceedingly frankly.
Evidently, when the phrase was first translated from Greek in the Renaissance, the Greek word for "trough" was confused with the Greek for "spade," and thus the modern version was born.

John Knox introduced it into English when translating a Latin text by Erasmus. "I have learned to call wickedness by its own terms: A fig is a fig and a spade a spade."

There is enough racism around without attributing it to inncous expressions