Beer: Also, you might be unconsciously lip-reading. (That doesn't work over the phone though.)

Bizibod: Maybe the earpiece of your phone fits the contour of your ear better when the earpiece of your glasses is behind your ear.