In early use desert was often used in the singular, and just desert might not refer to a punishment, but to anything that was deserved. In modern use it is typically found in the plural, and just deserts almost always is in reference to a deserved punishment, rather than a reward. And remember that just deserts has nothing to do with post-prandial sweets, unless it is that the punishment that you deserve is to receive none of these things.