"It never seemed to hurt the Vikings, invading Ireland, and elsewhere:)"

None of which could be described as neighbours, which may be why they chose them rather than the rest of modern day Scandinavia.