I have googled the phrase "is it legal for someone to withhold a British passport" Google preferred, for some reason, "is it illegal". And, as Richard suspected, it is. Except for the Crown, the government, or other authorised party. Which I suspect Voyages to Antiquity is not.
Should have googled first. And then signed "under protest". Except everyone on the ship was much nicer than the procedure, and it would have been rude.
However, if they pursue me over some trifling difference in the exchange rate between signing and making the payment, maybe I'll draw it to the attention of the BBC's Watchdog programme, which has had a previous interest in an earlier company with the same management, or the Guardian's money page, or Max Clifford.