I don't see any contradiction in the song. It's possible to be charmed by someone, to find them attractive, etc. but for various reasons one may not be in a position to pursue a relationship. The narrator is headed for Donegal, he doesn't live in the same area as the girl with the shawl, he may have a sweetheart somewhere else, he may be a rambler not in a position to court a young woman...Whatever the case, I've always liked the song.