Theoretically a "darby horse" would be a horse good enough to run in a derby, "darby" being the southern English pronunciation.

If Brown's texts were rooted in some English idiosyncrasy, a later singer might have found "darby" inexplicable as a horse and rationalized and relocated the word as a "derby" hat - a style described in 1864 as "all the go" in the U.S.