My understanding is that "Phantom" power is so called because dynamic mics can't see it and it has no effect on them. As long as you are running the dynamic mic in balanced mode there should be no problem.

Where you are most likely to come unstuck is connecting an unbalanced mic to a balanced input via an XLR plug/socket. (i.e. to connect an unbalanced mic or other source [with only two wires - screen and signal] to an XLR socket, you simply need to connect the spare XLR pin to earth. But don't do it if there's any chance that phantom will be turned on!