Hmmm. Don't know the origin of the phrase, but I'm in the U.S. and I don't remember ever hearing any intimation that it had racial connotations. "Getting down to the nitty gritty" is akin to "getting down to brass tacks" or "getting down to business." It does seem to have an earthier sense to it, but nothing racial I've ever heard.