When you start a sentence with 'actually', you usually are indicating that your interlocutor is wrong, or at least lacking in information. It may well mean "'You are wrong, and I am right, and you are at least a little bit of an idiot.'" So - what's wrong with that? Are you supposed to refrain from correcting others - or are you supposed to come right out and call them 'idiots' while you're doing it?