In some English Shrovetide dances, a couple were featured: a Holly Boy and an Ivy Girl. The holly represents the male principle, the ivy the female. In China, I'm told, it's associated with the mother goddess. On the other hand, Fun & Wagnalls Dict. of Folklore tells me that "in Christian lore, its evergreen quality symbolizes the immortality of the soul". In the carol, therefore, I'd say they have a sort of phallic significance. Remember that in mediaeval times the old beliefs were still rather strong, and Mother Church hadn't entirely purified (or emasculated) ancient pagan beliefs.