it doesnt even make sense because by definition both parties are semetic

No, the term was conceived to describe the hate against Jews, semantically you are correct concerning the term semites, but antisemitism explicitly means Jewish people.

Due to the root word Semite, the term is prone to being invoked as a misnomer by those who interpret it as referring to racist hatred directed at all “Semitic people” (i.e., those who speak Semitic languages, such as Arabs, Assyrians, and Arameans). This usage is erroneous; the compound word antisemitismus (lit. ‘antisemitism’) was first used in print in Germany in 1879[17] as a “scientific-sounding term” for Judenhass (lit. ‘Jew-hatred’),[18][19][20][21][22] and it has since been used to refer to anti-Jewish sentiment alone.[18][23][24]

Interestingly antisemites themselves coined that term, it’s not in the wiki quote but further in the article it goes into detail regarding that.