I'm no expert on this, but here's my opinion anyway.
Pops would have been upset about his daughter running off with any poor bum of a stranger, Gypsy or no. But by calling the rover a Gypsy, it's automatically to be assumed that he is indeed a poor no-good bum--after all, aren't they all?
Gypsies and beggarmen seem to be considered suspect, the difference being the beggar is being judged by what he does where the Gypsy is being judged for what he is, ethnically. It's not that the song actively disparages Gypsies, but that it repeats and accepts the prevalent prejudice against them.