According to the Guinness Book of Records I had as a kid, the two hardest are the Oboe and the French Horn. Having grown up with older siblings learning both, I'd like to add that they're pretty hard work for the listener as well as the player ;-)

The difficulty isn't necessarily down to the complexity of co-ordinating physical activities (that makes it harder to learn for sure) but also the nature of physcial activity- to play a brass instrument for instance, requires you to get and maintain exactly the right amount of tension on the lips otherwise you get all kinds of unpleasant noises and wrong notes. It also puts a huge amount of strain on the muscles, and a player needs to practice daily to build up the stamina required to play for a whole performance.