LH, you are looking at this from the wrong perspective. Humans, as a race are biologically degenerate elves. Similarly, homo sapiens has gradually interbred with and displaced homo neanderthalis, even though everyone can identify neanderthals and other trogolydites among us.

Elves have the ability to live, and live well, where we are not aware of them. As the literature frequently states, elves keep track of humanity and often take back children who have the potential to rejoin their stock. They leave inferior elves in their place to be raised among their degenerate descendants as humans.

These changelings do not really fit in well among the general human stock and frequently are referred to as "fairies" by those exogenous to their group. In general, they rarely breed and tend to eliminate themselves from the gene pool within one generation.