From sites I've found on him, James Weldon Johnson would have been about 21 when he wrote the poem about 'little brown baby'.
Here is some more info about Johnson:
In 1900, he wrote the song "Lift Ev'ry Voice and Sing" on the occasion of Lincoln's birthday; the song which became immensely popular in the black community and became known as the "Negro National Anthem." Johnson moved to New York in 1901 to collaborate with his brother Rosamond, a composer, and attained some success as a songwriter for Broadway, but decided to take a job as U.S. Consul to Venezuela in 1906. While employed by the diplomatic corps, Johnson had poems published in the Century Magazine and The Independent.

In 1912, Johnson published The Autobiography of an Ex-Colored Man under a pseudonym, the story of a musician who rejects his black roots for a life of material comfort in the white world. The novel explores the issue of racial identity in the twentieth century, a common theme in the writing of the Harlem Renaissance.click here http://www.poets.org/lit/poet/jwjohfst.htm