MGoH:
I don't think that making other people uncomfortable or fearful is something we should feel entitled to do if there's no good reason, or that "it is their problem, not yours".

If what we do makes reasonable people uncomfortable (walking down the street with a firearm or some such) then it is our problem.
Actions taken which are perfectly reasonable, and which don't make the large majority of people feel uncomfortable, may make some people who are at increased risk feel uncomfortable. In that situation they can choose to avoid putting themselves in that position. I still say 'their problem, not ours'.