Actually, even Hitler's 1939 extravaganza had plenty of positive effect. The best-remembered ten seconds of that entire Olympiad is Jesse Owens putting the lie to Nazi racism. And that's not something that is important only in retrospect, from our distant viewpoint 60+ years later: it was very widely reported and commented upon at the time, as well. If it did not convince huge numbers of former racists to change their minds, it probably had that effect of a few ~ and in any event, it did NOTHING to reinforce the opposite viewpoint.

If the Olympics had not been held in Germany that year, would Hitler not have invaded Poland? I seriously doubt it. The events of that Olympiad may or may not have affected the course of history; I believe that there were many relatively small repercussions, some good and some bad, but that the overall march of time would probably have proceeded in much the same way that it actually did.