There's nothing 'damaging' to the child if He/She starts at 4 or 5, as long as the child doesn't feel forced to do it.
That's why the choice of teacher is very important. The teacher must have good communication skills, patience, as well as good dancing technique; for it is in the early years that good and bad habits are formed.
Bren.