I think it's annoying because people who use it don't stop to think that they could just say "no" - a much more succinct and accurate answer!

What's so important about being succinct? Monosyllabic exchanges don't make for great conversation.   It's worth considering that there are not any words "yes" and "no" in either Latin or Irish...