I'm not a lawyer either (perhaps you should consult one) but, further to remarks made above:

If you are not a Canadian citizen, the Canadian Consulate is not for you; contact the Canadian Embassy. Consulates are for our citizens living or travelling in other countries; embassies are for foreigners who want to contact our government.

If you are a US citizen travelling by air, bring your passport, and passports for all your children who are travelling with you. If you are a US citizen travelling by road, bring your birth certificate and, if applicable, naturalization papers, and the birth certificates of all your children who are travelling with you. If you are neither a US citizen or a Canadian citizen, and you are crossing the Canada-US border, you need your passport and, depending on your citizenship, perhaps a visa. Contact the Immigration Officer at a Canadian Embassy to find out what applies to you and your children.

Canadian governments and, consequently, the agents you meet at border points are increasingly aware of child abductions. The issue is always what is legally required of you, however, and if you do not have a legally registered child-custody agreement that does not permit you to take them to Canada, that (at least) is not a problem. Mind you, your X2B could *make* it a problem, but he/she must get a court order, and get it enforced, to achieve that goal.

Sounds like you and your kids could use a break. Canada is a great place to take one. Happy travels!