Here's some of the information I found on the story from A Right To Sing The Blues: Jews, African Americans, and American Popular Song by Jeffrey Melnick:


****
pages 114-115

-snip-

"Irving Berlin's major biographer recounts that the songwriter was a self-taught pianist who hit only the black keys. Berlin referred to these as "nigger keys" and to his instruments as "nigger pianos."
(Berlin's lack of skill initially restricted him to the key of F-sharp major. Later he bought a transposing piano which could change keys with the switch of a lever below the keyboard.) The racist language----which biographer Laurence Bergreen says was typical of the heavily Jewish Tin Pan Alley---- is not surprising coming from a man whose early successes in music were accompanied by the dogged rumor that he kept a "little colored boy" in his closet (or basement) to write his music.

The anecdote offers a compelling narrative of Black-Jewish relations in which powerful Jews control hapless African Americans. The rumor also hints at an intimacy between the Jew and the African American which was unsettling for many observers.

In this anecdote, as in the Jewish use of the nickname "Nigger," we find a decidedly anti-utopian rendering of the connections of African Americans and Jews. Rather than the usual melting pot romance of Jewish involvement of African American music, the case of Irving Berlin's "little colored boy" emphasized a bleak version of group relations in which Jew and African American seem at once unhealthily close and yet estranged. The anecdote interprets Jewish interventions into African American culture as the result of theft and force rather than in talent and voluntary cultural merger. While the rumor imagined the Jew and the African American as physically intimate---- although it did not make clear whether Berlin was in the closet with the "little colored boy"---- it also suggested that the terms of the relationship were dictated by Jews, for the benefit of Jews. In the mythology of Berlin's "little colored boy", the Jew emerged as exploitative and paternalistic, and perhaps (most frightening) as sexually interested in the African American."

-snip-