The cheap plastics used in cheap plastic dinnerware continually emit odorous chemicals. When you eat with a plastic spoon, you get a whiff of some sort of polymer every time the thing comes near your nose. It's not really taste, but our senses of smell and taste are so closely tied that it may as well be.

And there are tactile issues as well. For one thing, plastic spoons don't conduct heat like metal spoons do. The natural warmth of your tongue warms a metal spoon so cold food glides off of it. Plastics don't warm as quickly so cold food tends to stick to plastic spoons.   

If it were a matter of all plastics actually affecting the taste, yogurt wouldn't be packaged in little plastic tubs, would it?