Steve you are right in that regard, generally just leave them alone. In the states the ground wasps or wood wasps can be a real threat to children. As long as they remain in the woods that is perfect. If they build a nest in your house it can be life threatening especially to small kids or adults that are sensitive.
In Florida the Mahogany wasp kills a number of people each year, mainly due to allergic reactions. They like, the wood wasp tend to nest in the attics of houses or under the porch. Aggressive critters with a painful bite.