Jerusalem in this sense "till we have built Jerusalem in England's green and pleasant land" is surely a symbol of perfection. But not anything much to do with the crusades specifically, I would have thoughts. I think there two intertwined bits of memory here. the fact that Jerusalem(specifically the temple) were destroyed, and there was(and is) a clear Jewish belief that it should not be rebuilt till some unspecified date in the future when the Messiah comes.
   The other relevant biblical thing is the exile of the Jews to Babylon, and the fervent desire to return to jerusalem "If I forget thee O Zion let my right hand lose its cunning". Coupled with other exiles(Egypt, the diaspora in Roman times) creating a general longing for Jerusalem ("next year in Jerusalem).
    Add in songs like "The Holy City"(Jeuralem, Jerusalem), place William Blake's Jerusalem, and you've got a pretty potent package.