Festivals would not have their atmosphere without a Fringe.
As you wander around Sidmouth or Whitby it is all those things going on everywhere that creates that feel of festival (nowhere more indeed than Edinburgh) and as Parasite says above makes it worthwhile for the host town.