Well, it is fairly common for non-religious people to base their estimation of religious people on the behaviour of the more obviously bone-headed of religious people, I suppose. Those ones are easier to notice. ;-)

The same is true of the more bone-headed atheists.

In both cases, they tend to go to extremes and be prejudiced unnecessarily. That makes them noticeable.

To put it another way, I've met idiots who were religious, idiots who weren't, and wise and gifted people of both persuasions.

Religion isn't the problem. People with a bad attitude are the problem.