I grew up near Shirley Avenue, Revere MA. Shirley Ave. was the center of the Jewish section of town. My parents bought a house square in the middle of the Jewish section. We were (for a while) the only Italians there.
All my friends were Jewish. I learned the song from them and, probably first heard it in 1951 or 1952, when I was 12 or 13.
It certainly was NOT considered anti-semitic. It was considered funny.