That is what happens, its a shame, but ultimatley people do have the choice of staying together and making a relationship work or spliting up. Where there are kids involved this often means that one of the parents will have little or no time with the kids, but that should be taken into account when they decide to spilt up. Life and love is not a film where we all live happily ever after once we have found Mr or Mrs Right. It takes work and commitment to make a relationship last.
If your ex's parents are going to tell your kids you are a horrible bitch who dumped their son for no good reason you may not wish your kids to spend time with them.
If your kids are not allowed to see thier father and you do not trust his parents you may not allow them to see the grandparents.
If you have sole custody of children you have sole reponsibility for them and the grandparents are after all the people who brought up their son.