The "debt" is not something owed to the taxpayers in general. It is owed to holders of bonds (including EE savings bonds) and treasury bills -- people who have essentially loaned the US gov't money with the expectation of getting a return on their cash.

This system, by the way, was invented in part by James Madison, with the end of preventing a coup. People who are owed money by the government, he reasoned, wouldn't want to overthrow that same government. Which is why it's comforting that so many t-bills are owned by overseas investors -- makes them less likely to declare war on us.

Alex