Steve, I just say "they're all on drugs" and leave it at that. It may be unfair but I don't care for most of the Olympic sports anyway.

Besides that, I feel that drugs are likely to have a bigger impact on things I don't enjoy (eg. running) than ones that might just attract my interest (eg. football).