FWIW, there's a Mississippi John Hurt song which has the following lines (as a counterpoint, and why Leadbelly maybe stayed away from 'yeller gals' doors)

"Some crave high up, I like black and brown.
Some crave high up, I like black and brown,
Black won't quit you, brown won't (lay you down?)"

possibly a feeling that those who could 'pass' wouldn't be loyal to those who couldn't.