"It was full of 'English' Anglo-Saxons"

You're absolutely right on that. The south and central Scotland was a mixture of Anglian Northumbrians and indiginous British speakers (Old Welsh for want of an easier description) whilst most of Scotland to the north was Pictland who mostly spoke a language closely related to the British speakers to the south. Gaelic was spreading from Argyl and at some time took over from Pictish. In most of the Hebrides, Northern Isles and far north of the mainland the Norse language was to the fore. It was a hotch potch and nobody spoke like Billy Connelly. Though the Anglian's language was an ancestor of how he speaks.