I find that you decide, using whatever info is available.
If someone remains silent then your decision can only be based upon appearance, expression, body language, and maybe aroma if they get close enough. A call centre worker decides just by a short conversation as no other data can pass.
What seems curious is that having made an initial judgement on very little acquaintance, it takes a long while for any revision of the judgement to "take", somehow the first opinion seems very resistant to change. Love (or hate) at first sight, anyone?