The James Brothers are mentioned in True Grit by Charles Portis - 'Rooster' Cogburn's backstory is that he rode with Quantrill's Raiders. In a scene in the novel where Cogburn re-educates a couple of boys who were torturing a mule, he tells them to mention to their father that (if memory serves) "Mr James, a bank adjuster" has called, and so on. Great book, by the way.

The James Brothers might have particularly despised Benjamin Butler because of his noted anti-Semitism. I see an online query from one of the Briscoe family (an Irish Jewish family) who says that they are reputed to be related.

Given the persistent legends of the James brothers, and other bank robbers, helping out poor farmers that the banks were screwing, it would be an interesting project for some sociologist to collect family stories of this.

People - at least, many poor people - saw the outlaws as heroes for taking on the avid, all-powerful banks.

The nearest thing to the hatred of the banks in rural America at that time is the modern feeling about vulture funds, which buy up house loans en bloc and then evict the mortgage-payers and sell off their homes for tiny amounts.