I still reckon it means the fair town of Scarborough. The second line of the first verse says 'remember me to one who lives there' which I assume is Scarborough itself, not the fair or market. It's rather like lines mentioning a 'lady fair' or 'maiden fair', e.g. as in When first I came to Caledonia.