This is how I've always understood American usage:

"Pants" is a non-specific term referring to any type of leg covering other that skirts or kilts. "Trousers" is generally used to refer only to "dressier" varieties of pants. The pants worn with a suit may be called "trousers", but bluejeans never are.

Bruce