The reason why i think it's rude is because I was brought up to beleive that taking your hat or cap off indoors was being polite. And Keeping it on unless for religious grounds then it's rude.