In my experience as a caller, experienced bands generally know how fast to start a tune and can adjust their tempo if the dancers need it. Adding to Flora's suggestions, it's generally a lot easier for bands to speed up than to slow down, so if in doubt they tend to start a bit slow.

Less experienced bands quite often ask me to set the tempo, which I can only do by galloping around the stage a bit. I like to hope it provides extra entertainment, as well as an incentive for the bands to work out their own speeds fairly quickly!

Some of the bands I work with have drummers, and others don't. Of those which do, I can only think of one where the drummer sets the tempo - generally there's one melody player leading, to a greater or lesser extent.