Try to keep in mind the fact that the boy is HIS child too. I know that's difficult for a Mother to do, but it will help to ease your mind. It's not like he kidnapped the boy.
I don't know what state you live in, but you may want to consider that if the boy lives with his father, you could end up paying child support.