This song is now widely sung as part of the Cornish pub "Shout" revival.
I have done quite a bit of research, not into the origins of the song, but the possible origins of the folk lore that surrounds it.
The most famous robber of the Lizard is probably John Carter. Porth-en- ayls or alls (AKA Prussia Cove) is a short distance from Cadgwith and all the fishing communities are interlinked. I have a feeling that the Robbers Retreat, might have been some kind of folk memory about The Carter Family. I have extracts below that would explain the lines: "Caves in Yonder Mountains"

and "Lay drooping his head"

However, there is no direct reference to "The Beauty of Cashmere or Kashmir" It seems very odd that if this refers to a flower, then why is it followed by the male pronoun? Flowers are generally considered to be Female in gender.
It is possible that Kashmir or Cashmere was possibly a common term for a smuggler (see below) in the same way that Corsican became connected with Bandits.

From "Smuggling in The British Isles" by Richard Platt unfortunately out of print.

The following extract about Kings Cove or Prussia Cove


..'so sheltered and secluded that it is impossible to see what boats are in the little harbour until one literally leans over the edge of the cliff above; a harbour cut out of the solid rock and a roadway with wheel-tracks, partly cut and partly worn, climbing up the face of the cliff on either side of the cove, caves and the remains of caves everywhere, some of them with their mouths built up which are reputed to be connected with the house above by secret passages — these are still existing trademarks left by one of the most enterprising smuggling gangs that Cornwall has ever known'

The following extract about John Carter (the King of Prussia) being nearly taken by the crew of a Man-o-War after his own friends deserted him.

He was struck down, severely wounded, and left for dead, but after several hours his body was still warm although 'his head is all to atoms' as one of the guards observed. Despite his injuries, he was able to crawl across the deck and drop into the water. Once in, he found — not surprisingly — that his stout swimming skill had deserted him, and he was forced to pull himself along ropes at the ship's side, until he could touch the bottom and crawl out of the water. On land, he was picked up, half dead, by local men...

Carters own Diary about this incident says that his nose was nerelt severed, hanging on a piece of skin and that the top and back of his head was deeply cut by some terrible cutlass slashes.

Britannica entry on Smuggling in India in the 18th and 19th Centuries.

"Attempts by the Chinese government to stop the smuggling of opium led to the opium war of the 1840s. British India in the 19th century suffered smuggling of salt between states with different tax rates, while smuggling of all kinds of dutiable goods occurred between Goa and India"

Kashmir is also on the Silk Road and passing through the Himalayan Valleys probably produced quite a lot of contraband for robbers.

It is possible that Kashmir was a kind of nickname for a Smuggler.

I rest my case!