It wasn't betting with a bookmaker. Two wealthy racehorse owners would simply put their stakes in the hands of a third party. Their horses would race against each other and the winner would take all.

At Newmarket the Jockey Club established a dominant role. Many of its races were high-stakes matches made between two wealthy owners. Such wagers potentially combined racing rivalries, power politics, social rank, informed sociability, and the social display of the match-maker’s prowess, wealth, status and masculinity, as he demonstrated his judgement about the horse’s qualities relative to the one he had chosen to match it against.

Not all of them were honest. When Daniel Defoe visited in the 1720s, he was shocked at the way the nobility and gentry were ‘busy in what is called the sharping part of the sport, of wagers and bets’, trying to ‘circumvent one another and […] pick one another’s pockets […] to the indelible shame of men of rank and quality’. [digitens.org]

In a famous match with Mr. Cookson's Diamond over the four-mile Beacon Course at Newmarket on 25 March 1799, Hambletonian, ridden by Francis Buckle, won by a neck in a time of 7 minutes 15 seconds. He is said to have covered 21 feet in a single stride at the finish. Sir Henry [Vane-Tempest] had wagered 3,000 guineas on the outcome.

The horse was afterwards the subject of the painting, Hambletonian, Rubbing Down, by the great equine artist, George Stubbs, who was then 75 years old. In 1800 Hambletonian won his only start in the Great Subscription Purse for six-year-olds and over at York. [Wikipedia]

Harry Ogden became the first-ever bookmaker in 1795 when he laid the odds and took bets at the Newmarket horse races. Before Harry Ogden, however, the types of wager were very different from what we have nowadays. For one thing bets would generally be offered on one particular horse (the Favourite), against the field. There being only two outcomes these were generally even-money bets, and the wagers were made between individuals rather than being administered by a `turf accountant’. [telescoper]