@JPhil,
You're comparing apples to oranges here (to stay within the fruit group).
The color blue is blue because you're taught it's blue - you'll draw from your memory when you see something blue, or any other color for that matter.
If your mother would have taught you that blue is actually called yellow, you'd assume the name for blue is "yellow".
As with food items (banana) you will know when you've eaten a banana before. Lets say, we blindfold you and let you taste a banana - you'll know right away what it is because you've eaten it before. Now keep your blindfold on longer and we'll let you taste an exotic fruit that you haven't tasted before - you will not know what it is, nor will you be able to tell from smelling it.
Now you can take your blindfold off and realize that your senses teach you what's what.