I remember RH being asked about this during an interview. He did seem to be apologetic about it, explaining that at the time (50's 60's) it was just the everyday attitude to aboriginals and not done with INTENTIONAL malice.
In much the same way as my old mum would say n***** brown
Or eeney meaney miny mo....
Or "dark person" in the woodpile (that was a strange one)

Most/some of us nowadays can see that this kind of talk is hurtful and best avoided.