Mark,

That line is difficult to decipher from the recording. At first, I thought it was something like the line you have posted - I know a similar line from 'Ain't No More Cane on the Brazos'- but then I wasn't so sure. My hearing of it was influenced by someone's interpretation in a previous thread on this titled 'Eloise'. However, on further multiple listenings, I reckon you are right, even though the first part is terribly slurred. It could well be:

'But they didn't drive the women like they drived the men'.

Thanks, Stewie.