There is little doubt that the banjo is a descendant of instruments used in Africa. An instrument in the hands of slaves consisting of half a calabash, covered with skin and with a long neck, in mentioned toward the end of the 17th century. By the late 18th century, the banjo as we know it had evolved into the manufactured instrument we know today, probably with the expertise of men like Boucher, as suggested by Butch. These facts are synthesized in Dena J. Epstein, 1977, Sinful Tunes and Spirituals, Univ. Illinois Press, the book which perhaps says most in one volume about the early instruments of the slaves. (The development of the banjo is not discussed, but she reproduces a painting, late 18 C. from SC in which a black musician is playing a very modern-looking banjo, so the instrument we know had evilved by the 1780s). Drums were frequent in the earliest days, but were outlawed in many areas.

We think of the Irish-English-Scots influence because these immigrants were or became English-speaking. Many Irish landed at southern ports from the Carolinas to New Orleans before the Civil War, but Germans and central Europeans were also influencing the music; the tunes are there, particularly in the dance music.
Butch is correct when he mentions that blackface portrayals go way back in European culture. Rather than Celtic or English, "European" is perhaps a better term for immigrant musical traditions.