The OED doesn't explain the derrivation, but it sure give a lot of examples for "skin off ..." Skin off my bugle?

no skin off one's nose and varr. (colloq.), a matter of indifference to one. 1920 S. LEWIS Main Street xxv. 312 Go to it. No skin off my ear, Nat. Think I want to be fifth wheel in the coach?
1926 Mantrap viii. 95 If you think..that it's any skin off my nose to lose the pleasures of your company..you got another think coming.
1930 Amer. Mercury Dec. 420/1 It ain't no skin off of Hymie's bugle.
1934 J. O'HARA Appointment in Samarra vi. 158 Okay. No skin off my ass.
1938 D. BAKER Young Man with Horn I. iv. 30 It was no skin off Jeff what color his old lady painted the piano.
1955 A. MILLER View from Bridge 102 Don't thank me... It's no skin off me.
1960 D. LYTTON Goddam White Man v. 113 But it was no skin off my nose that she was dead.
1963 Australasian Post 14 Mar. 51/2 If you want to yap on like a drongo in the DTs it's no skin off my bugle. Go ahead: be a gig! See if I care!
1966 J. PORTER Sour Cream v. 60 Our arrival was no skin off her nose and she didn't pay all that much attention to us.
1971 B. MALAMUD Tenants 35 Make it like eight [o'clock] or around that if it's no skin off you. If I miss a day don't fret on it.
1972 R. MILNER in W. King Black Short Story Anthol. 378 Then Clyde said it was no skin off his ass.
1978 L. MEYNELL Papersnake xiv. 188 It was no skin off my nose... My heart wasn't hurt, even if my pride was.