From the M5 you've either got to go through Taunton or over the top of the Quantocks. Either way, it's a bit more than 20 minutes.

Never mind, Halsway Manor is lovely and should be an eminently suitable venue for what you're doing (there's even an ancient gargoyle over the front door with a finger in each ear!).