The naval expression skin off my/your back refers to flogging. So 'that's no skin off my back' means 'I won't be flogged/punished for that'. The boxing idion seems right for nose.

When I was a little boy (oo, haven't you got a good memory, Les!) if I asked my grandma what she was baking, she would always say 'whim-whams for ducks to chase at!' I never heard that anywhere else.

Les