According to this article, the grandfather had three wives, but it says that Sarah was the one who actually raised Barack's father (his father's natural mother left when her husband moved the family to another part of Kenya). Barack considers her to be his grandmother, and she is the one he stayed with when he went to Kenya, and he calls her his grandmother.