That should be:

"You won't worry so much about what other people think of you when you realize how infrequently they do."

Forgive me Mr. Wallace