TO David Miller: As another possible origin, I've heard a few people state it was "to see FINNES lady" (not 'a fine' lady), and checking on google gave me a page saying: "Who she [the 'fine lady'] really was is a matter of controversy. Some say she was Queen Elizabeth I herself, others she was Lady Godiva. Another probable theory claims that she belonged to the Finnes (pronounced as 'fains') family, the lords of the nearby Broughton Castle."

Interestingly, another page on the same site gave:
"One of the possible interpretations of the poem identifies the 'fine lady' as Queen Elizabeth I, who travelled all the way to Banbury to see the newly erected enormous stone cross.

"While the rings obviously refer to the jewelry a queen would wear, the bells probably refer to the fashion of wearing pointed shoes with bells attached favoured by the nobility of the time.

"As Banbury was situated at the top of a steep hill, a white cock horse (a large stallion) was provided by the town's council to help carriages up the steep slope. When the queen's carriage was going up the hill, one of the wheels broke. So the Queen mounted the horse and made the rest of the journey on horseback. The people of Banbury had decorated the horse with ribbons and bells and provided minstrels to accompany her, hence the 'music wherever she goes'."