Agreed, it's a tricky concept. The point is that there's no bisecting of the time period (day before plus day after) except for those points on a longitudinal line where the first day starts exactly twenty four hours before the instant of solstice so that the second "half" of the time can be exactly twenty four hours after the instant. If less than exactly a full day is used up getting to the point where the solstice happens, then part of that same day is spent going away from it. The "following day" spends all it's duration moving away from the point where it happened, so at the end of the second day you're further from the point of maximum solar exposure than at the start of the day before.
See - it's really very simple.
JD -
The eccentricity of the earth's orbit, if I remember ancient studies, is only on the order of about 1 percent (0.01). The orbit really is very nearly a circle. I'm sure with accurate enough instruments one might find an effect of eccentricity, but that's not the main explanation. The difference between when earliest sunrise and latest sunset occur is due, in simplest terms I can think of, to the fact that the length of day is changing during each day and while it's pretty trivial rate/interval calculus, it's nowhere near as simple as for BPL's confusion, to explain this one without drawing pictures and writing equations. And I don't have a curly delta or a swoopy ess sign on my keyboard. (Which makes it easy for me to pretend I might remember how to do it.)