"It seems pretty obvious to me that the rich taking from the poor is selfish, while the poor taking from the rich is less so."

Have you really got nothing else to think about?