In the US, as is usual for almost anything dealing with laws, there are some variations for each individual state, but forcing you to go somewhere "without cause" would generally violate at least three clauses of the US Constitution, and most State Constitutions replicate the principles there.

Usually you could "decline," or perhaps request that you be allowed to come in later at a more convenient time; but the police could possibly appear later with a warrant issued by a Court (one with sufficient authority) ordering you to appear, and/or that you be taken into custody pending an appearance in court. With a sufficient court order (warrant), the police might be able to hold you (as a material witness?) without technically placing you under arrest.

Your response would need to fit the situation and the persons (police?) involved, and usually reasonable cooperation is best. While most "big city" police should know the rules better than you, and should be expected to observe them, some "small-town cops" may be ill-informed and more akin to street thugs than to law enforcement officers (fairly rare?), so you'd want to do what allows the best chance for survival with minimal harm to yourself.

John