I definitely heard it as "down" not "town" and puzzled over it initially, as Scarborough is north of Yarmouth, but apart from any consideration of prevailing winds, it was explained to me that going back to your home port, in this case, was "going down home". The UK's prevailing wind directions vary a lot by the way: can turn right round in a deep depression, as happens in big storms!
Although William Delph (Delf) was from Grimsby, Scarborough would have been roughly in the same direction if you were heading from Yarmouth. This interpretation may of course be pure speculation. ;-)
And I looked up the thread in anticipation of my own "three score and ten" coming up shortly! :-)