Most of the time there is going to be some fair trade operation, co-ops and so forth, who are producing the same kind of stuff, but made and processed by people who are being given fair conditions.
By buying from the exploiters in effect you are discriminating against the fair trade people.
There's no perfect answer - but in a way it's analogous to supporting a strike, even though you may know that the blacklegs are likely to be desperate people who didn't feel they had an alternative to scabbing.
Solidarity for ever.