The colors we see as "prototype" (best example of) are, yes, determined by the physics of color perception. even people with one word for "either blue or green" will pick a standard blue as the best example, not the aqua in the middle.

But whether we see 2 wavelengths as "different shades of one color" or as "different colors" is linguistic, not physical.

It's the same as whether we hear different vibrations as the "same" speech sound or not. Depends on whether the difference is phonemic (meaningful) in your language or not. English speakers can tell the difference between the P of POT and the P of SPOT only by putting their hand in front of their mouths and feeling the difference in air puffing - but speakers of languages which differenciate the aspirated and unaspirated P the way English distinguishes between P and, say, B, can actually HEAR the difference. We anglophones just hear them both as P, despite having the same inner ear workings.

Way cool!