I don't think that your hypothesized situation is specific enough.
a person walks up to another and greets them. The response is non-verbal.
Walks up to stand face to face? Greets as if to open a conversation? Or just just says hello as they walk past? Co-worker, friend, aquaintance, stranger?

If I were to pass a stranger who greeted me, I might respond with just a nod. If a co-worker came up to my desk and appeared to have a question, I'd respond with just the quirk of an eyebrow. A friend I would throw a smile to when passing by, but would greet verbally if we were stopped.

I think in answer to the spirit of your question, a non-verbal response is not necessarily rude, but may be in certain circumstances.
A lot depends upon the perception of the receiver of the message.