From the Great & Glorious Wiki...

The expression is said to have originated from poker, in which a marker or counter (e.g., a knife with a buckhorn handle during the American Frontier era) was used to indicate the person whose turn it was to deal. If the player did not wish to deal he could pass the responsibility by passing the "buck", as the counter came to be called, to the next player.