Depends who says it and their tone.

Identity is important and if somebody questions your identity based on false criteria it can be offensive.