Not everyone voices their opinion. They imagine your discomfort based on their own unease at praise and as for the reaction not liking something they sqirm. So they say nothing. I found the audience come with a preconception of you or the night. I remember two clubs where I started as s joker in the trad club and as a very serious poet in the less stuffy one and had more difficulty doing the "other" material in the venue thereafter. It was easier in the latter to change their expectation of me.