I did a google and found this bit of info at The Northwest Durham and Derwent Valley History Site


Despite poor and dangerous working conditions, low pay and long hours, the often tyranical coal owners of the last century would not hesitate to resort to such measures as eviction to deal with miners' strikes. The `candymen' employed by the coal owners to evict the miners were disreputable characters of the lowest order, brought in from the docksides of the large towns in the region. Described as "low, mean ragged fellows", the "yelling, shouting, and tinpanning together with the pitiful cries of children had no effect on these inhuman beings employed to do this work".

Doesn't say why "candymen" name would be linked to a "mean and ragged fellow" though.