I do find the descriptions of winds confusing. For example, the north wind comes from the north, which means it will be cold (as in 'The North Wind doth blow, and we shall have snow'), but if I were to set off in a northerly direction I would be heading towards the north.
Incidentally, unlike Steve Shaw, I wouldn't think of being in Devon as 'down in Devon', but I would think of being in Cornwall as 'down in Cornwall'.