IMO, the question is stated too broadly to permit a simple answer.

I care a great deal what the few people I respect highly think.

I care somewhat less what the people I respect, but less highly, think.

I don't give a **** what the people I don't respect think, unless it impacts my income or they think so little of me that it puts me in physical danger.

Peter.