So how would it differ from the statistical expectation that, being English, you probably don't go to church very often?

The guesses we are likely to make about other people, on that kind of basis, is just part of the way we navigate our way through life. I can't see there's any reason for us to get upset about them.