MtheGM:

Every point on your body's front is directly reflected in a flat mirror straight across from its physical location. Imagine the points of the body connected to their reflected corresponding points in the mirror. The ones on your left shoulder would go straight over to the side of the mirror on your left. The ones on your right go straight over to the side of your mirror that is to your right. This is the simplest possible scheme of straight-line reflection.

For your mirror image to be "correct" from your point of view of it's point of view all those straight-line reflections would have to cleverly find their way to the opposite side. Very complex, improbable behavior for one ray let alone millions of them.

Just trace the rays and it will make sense.