Vermiculite insulation can certainly be nasty stuff. I'd certainly have a professional inspect any such installation before trying to remove it.
I also worry about the long term consequences of spraying fiberglass insulation into attics, and inhaling their fibers. I'm less worried about fiberglass installed in batts.