Rumncoke has it right, but perhaps a bit more explanation might help. On a square-rigged ship, the sails are stretched along the yards (or yard-arms) and dangle down from them. The angle of the sail to the wind is then controlled by turning the yards and the sails are tensioned by means of ropes (sheets) to the lower corners.

One thing I hadn't realised until I did some sailing on tall ships was that the lower yards are turned further off the wind than the higher ones, so that if you consider all the sails on an individual mast as a unit, there is a distinct twist in it, like the wing of an aeroplane. This explains why tall ships can move, even if the wind isn't directly behind them.

As the sails are braced round, the angle of the topsails to the wind is smaller than the lower sails, and so when the limit is reached - when the sails can't be turned any further without being back-winded - the topsails start quivering first. So bracing the yards around until the topsails just start quivering, then easing a trifle until the shivering stopped, meant that the sailors would be getting the best possible thrust out of their sails. A quivering sail isn't efficient.

"Shivering the topsails" was also used deliberately when the ship needed to slow down without going to the laborious task of furling all the sails. A back-winded, shivering sail is creating drag, not thrust and does the job nicely.

How this all fits into the use of the phrase in songs varies with the nautical expertise of the songwriter and/or the audience. The sails wouldn't quiver as a result of anything the river would do, so that's probably poetic licence, but the Williams quote above sounds authentic to me.