Depending on where the person lives and works, I should think male to female is the most difficult, since (according to feminists) many women have a harder time in society than men. If a person born a male has become accustomed to enjoying the slightly more privileged life of a man, they'd find things a bit different once they become a woman.
There would be a subtle power shift, and they'd need to adjust to a slight change of treatment in some circumstances.