From the depths of (very old) involvement in electronics I would say the problem would be the loading of the output stage of the amp ! It would depend on what the output on the amp is , but it would only be a
problem at 'full' volume on most amps . The imbalance would not help the sound , and I would rather put both speakers on for safety's sake . If space is a serious problem just put one speaker on top of the other !