It doesn't take hundreds of thousands of years for traits to be selected for. Nature is red in tooth and claw. If the only work available to you is say weaving, and you starve and your children and starve because you are too clumsy to do any weaving, then nature...well economic nature or social brutality or whatever..has eliminated all the clumsy fumblefingers because they starved before they reproduced or were done reproducing and their children starved with them, unless, the children supported the family or whatever.

If you were too short-sighted to hunt and that was your only source of food, say in the Arctic..nature would eliminate the short-sighted from the gene pool (assuming both men and women hunted).

Nature can work pretty fast in terms of epidemics, some people selecting for running over others, i.e. Huns...and the ones without the aggressive traits lost forever...people not being able to survive famines without certain traits...this does not have to be a slow process...