Remembering the Santayana quote: "Those who cannot remember the past are condemned to repeat it.", removing statues that 'glorify' slavery (whether they do or not) without replacing them with something equally memorable seems to risk coming to a state where slavery is not a past memory, but something which has never been considered.

Leaving the statues in place at least allows them to be discussed.