Soley from personal experience, it's not the nail polish that's the killer, it's the remover -- even the "moisturizing" kind and so forth. But for me, nail polish tends to flake off and take a slim layer of nail with it.
My nails grow extremely fast, but they are very thin and flexible. I used to spend a small fortune on my nails at the manicurist, erroneously believing regular care would strenghten them. My manicurist finally told me to go home and leave them be, no polish or anything. Sho' 'nuff, within a few months I had a nice set of (still thin and flexible) healthy nails.
One warning about silk wraps -- they're ugly unless you coat them well with nail polish.