Most of the collected versions of Child 2 don't mention Scarborough at all, having an alternative or unspecified location. The only ones with verses about 'Scarborough Fair' seem to be those collected in Yorkshire - i.e. near Scarborough.

As for "the whole point of the song", this becomes more and more obscure as you get away from the 'Elfin Knight' variants (which are those most closely resembling the oldest available source version), until by the time it evolves into 'Acre of Land' it's lost any sense of a dialogue and has become a nonsense song.