The offenses outlined by President Madison were a result of The King's Orders in Council of January 11 and November 7, 1807. They were intended to restrict by neutral nations with France(and were just the sort of thing we enthusiastically advocate when used against, say, the Russians).

Since the US was the major Neutral Nation, we were the chief parties who "suffered injuries and indignities". Add to that a feeling in some quarters that the British were encouraging Indian attacks on American frontier settlements, and the above mentioned
desire to incorporate a bit of British and Spanish territory into our Union, and you have
enough votes to drag the country into war.

The Orders were actually repealed in June, 1812, seven months after Madison delivered the above speech, and a couple days shy of the American Declaration of War. So we went to war after the "casus belli" had been resolved.

Worth reminding all here that our most decisive victory, the Battle of New Orleans,   was fought on the 8th of January, two weeks after the peace treaty had been signed. And so it goes...