To me, the long white robe that fits too soon is another tongue-in-cheek reference -- this time, to the tendency of people to put on weight as they grow older.
The narration is done by an older person, clearly. That's why the banjo is now just hanging on the wall -- ran out of energy, hands arthritic, whatever. And "the old grey horse that I used to ride" is another bit of nostalgia for the days when the narrator and the horse were both younger and livelier.
The robe doesn't imply a female singer. Everybody in heaven got a harp and a robe in those days. And men as well as women wore white kid gloves on formal occasions. Again, few African Americans could afford them, so the reference in the song is to symbolic kid gloves, ie they are a symbol of something fine to be worn to a special place.
Even the "sweet sixteen" doesn't really imply a female, although it struck me that way at first too. It implies innocence, a lack of worldliness that was the right way to approach the coming trip to heaven. "Except ye become as a little child," etc...
And I don't think women wore ulster coats, which I have always taken to be a heavy form of overcoat.
Okay, I don't want to beat it to death, but I have been singing this a lot lately and I had to think about the words because there are just a couple of places it has to be brought up to date. This may outrage some people, but I changed "we'll telegraph the news to Uncle 'Bacca Juice" to "we'll telegraph the news to Uncle Willie Bruce," because the original was just too much of a negative stereotype for my taste. And sometimes I sing "overcoats" instead of "Ulster coats," but aside from taking out the dialect I think those are the only changes.
The dialect was actually more singable, by the way. Bland had a genius for matching lyrics and tune.