Cutting away the main mast would be a rather too permanent way of bringing a ship to a stop.
"Brailing up" is to pull the sails up to the spars and tie them up there with little ropes called brails- like the final stage in reefing them, depriving the ship of any drive from the aft (mizzen) mast.
Lowering the topsail(s) and brailing up the mizzen (which is the line I know in Henry Martin, does make sense from, a sailing point of view, since the captured ship would still be able to manoeuvre, but would not have the power to escape because of the reduced sail area.
The next phrase "bring your ship under my lee" means to move the ship close in to the down-wind side of Henry Martin's ship, again putting it in a position from which it could not sail away since the pirate ship would be "taking the wind from its sails"
Needless to say, cutting down the main mast would make this manoevre impossible- and I've never hear of the phrase "throw over the topsail" and I can't think of what it could possibly mean!
Cheers
Dave