Actually, it used to be a non-issue about removing your hat indoors. Up until about 150 years ago, NO ONE did. one removed one's hat for a lady, or important person, then returned it to one's head. It wasn't rude to take it off indoors, just not done. Things may be changing again.