Having thoroughly checked my Shakespeare concordances, etc, I can tell you Shakespeare never said anything about a fingernail moon, or used the metaphor, at least not specifically as such. He refers repeatedly to a horned moon, though, which was a more common image of the day - both things hanging on the horns of the moon, and a character playing the horned moon in MSND.
I suspect, like Mrrzy, that it's just a relatively common metaphor because it resembles the curve of a nail. There was another story I'm remembering for childhood about a princess who wanted the moon, who said it was just the size of her thumbnail, she could tell because if she put up her thumb against the moon, it just covered it. So the court jester made her a little silver disk the size of her thumbnail to hang on a chain from her throat, so she could have th moon. Another lovely image.