That's because, the worst thing you could call a highlander was "Sassenach", which equated to raper/pillager. Down through the centuries, it has come to mean the English.