Because it was in instrument of black people and then of working people esp. Southerners. It's a simple case of bullying the less powerful. (Nobody plays banjo at the symphony.)

I like banjo.