It's great that you are now free of work and able to spend your time doing more interesting things, and I do know what you mean about the "colleagues yakkity-yakking all day about fluff" - I remember that from the time I spent employed instead of working for myself.
But when people said that they hoped my day would pass quickly I used to find that upsetting. That day was a day that I would never have again, and however dreary and uninspiring the way I had to spend it, I refused to wish it away.