I've got a faint memory of Peter Davidson, the Doctor after Tom Baker, wearing some sort of scarf, but I'm not even sure about him. I'm pretty sure none of the others have had a scarf.