Steve with some farms in the UK making over £1,000,000 from the single farm payment and non compliance putting at risk some or all of that payment the risk of being caught in a spot inspection is very real.
Spraying regulations have been tightened dramatically in order to reduce environmental damage. If you see a contravention there is nothing to prevent you reporting it. A few seconds of video clip of vegetation swaying in the wind would enable anyone to assess if conditions were conducive to spraying, as would vegetation dieback in your garden.
I spent some years trying to get a hawthorn hedge established on my boundary surrounded by by successive crops of rape and wheat. I had thirty % dieback to what I can only assume to be toxic sprays. The failure rate was too high to be anything else.
The revised regulations are now in force. If you think you are getting sprayed you can insist on the COSHH sheet for the chemicals used and make noises as to why you were not given prior notice. It is no longer a free for all. You could always stir up your local defra, I am sure a man of your wit could generate sufficient grief to force compliance in your area.
Note the date at the end of the second link when opened (effective 2016)

http://www.hse.gov.uk/pesticides/topics/using-pesticides/spray-drift.htm

http://beefandlamb.ahdb.org.uk/prepare-for-new-pesticide-regulations/