Anti-Semitism is a phrase that was coined by someone who was himself an anti-Semite, according to his definition of the term (someone who is anti-Jew). He coined it to use as the name of a movement he was starting. So its history has always been in reference specifically to Jews.

My guess is that he used the term "Semite", even though it is inaccurate to use that term to only refer to Jews, because he was probably a very ignorant person who needed an impressive and official sounding term in an effort to help make his movement look more legitimate than it deserved.

The term has been rendered almost meaningless by people applying it to those who are not anti-Semites, but who are critical of the government of Israel.