One I'm guilty of using, and always get cross with myself for, is "D'you know what I mean?"

I keep using it as a filler, potentially to elicit some form of attendance indicator. It's fine if the person you're talking to is looking confused, because then they've got the chance to say "No", mind you.

"At the end of the day" annoys me as well. No idea as to why.