One should remember that "buck" is the regular name for the male of many animals, and so gets applied to male human beings of races or other groups that the speaker regards as inferior. For plenty of examples of this nasty habit, see the OED under the first "buck" entry.