But they are totally empty threats. You've informed them of the truth, and that's the end of it. They can huff and puff, but it's all a bluff.

You could send the threatening letters on to your MP and ask him or her to do something to stop them harassing you - it's the kind of issue MPs like, because it doesn't involve politics and can get a bit of press attention. And lowly bureaucrats tend to get in a real panic when they receive a letter from an MP.