True, but the slaves themselves had a definite pecking order established as to who got to order who around within their own ranks. With luck, a slave could become a free man in time. If so, he could soon hope to afford to have slaves of his own. ;-)
Like I say, it was a moral blind spot in their society.