I once lived in a house that had been owned by some unpleasant people. It wasn't just the white walls and sand colored carpet that they left behind. They left an impression on the house. I could feel the anger and the resentment they had left behind.
Your grandparents left their mark on the house. From the sound of them it's a good thing. The next owner may treasure the swing set, and the other things your grandmother left behind, and while I now sound like a new age kook, your family left an impression on the house that will stay after they are gone.