The millennium edition of Brewer's Dictionary of Phrase & Fable contains the following entry:

""Have no room to swing a cat". To be in a restricted or cramped area. There are various suggested origins of the phrase. Swinging cats by their tails as a mark for sportsmen was once a popular amusement. Cat was an abbreviation for cat-o'-nine-tails and in view of the restricted space in the old sailing ships where the cat was often administered, the expression is more likely to refer to this kind of cat. However, cat is also an old Scottish word for rogue, and if the derivation is from this, the 'swing' is that of the condemned rogue hanging from the gallows.

"At London I am pent up in frouzy lodgings, where there is not room to swing a cat." (Tobias Smollett: The Expedition of Humphrey Clinker, ii (1771)."

The Oxford Dictionary of Phrase & Fable mentions only the cat-o'-nine-tails version.

I can't find any reliable source that mentions manoeuvring a boat or ship in a confined space as the origin of the phrase, but nobody seems certain enough to come down firmly on the side of one or other of the suggested origins.