It seems he may have grown less moderate than he used to be. I am not that well-versed on what you have both referred to, but the following sure makes him a conservative in my view.(Please note, I am NOT responding as a means to incite a debate about Specter; only defending my use of "conservative.")

Specter: March 18, 2003, sided with the Republican wing of his party and voted against the "Patriotic Pause" amendment which would have delayed tax cuts in favor debt reduction protecting Social Security and spending on the war, homeland security and economic stimulus [Vote 57, March 18, 2003].

Specter: March 18, 2003 sided with the conservative wing of his party and voted against an amendment that would have eliminated most of the tax cuts in favor of shoring up the Social Security Trust Fund [Vote 58, March 18, 2003].

Specter: March 25, 2003 Specter sided with the conservative wing of his party and voted - against his moderate leanings - to support Bush's irresponsible tax plan. Specter voted against an amendment to reduce the Bush tax cut to $350 billion and dedicate the difference to a reserve fund to strengthen Social Security. [Vote 93, March 25, 2003]