I wouldn't have thought there was a problem as long as you are careful.

One thing to watch though is the impedance of the speakers and amp. If your amp required a min of 8 ohm, it might not take too kindly to being connected to 4 ohm speakers.