I think it is because when two people are talking directly to each other, they know that they can be heard even though they speak softly. People talking on the phone tend to speak as though the phone isn't even there and they must shout to the person on the other end. Also, disembodied conversations just seem weird. Reminds me of the characters I saw when I was working downtown.

jeffp