Never heard of Area V4 (insert alien joke here), but I'm gathering from context that it's in the brain rather than in the eye, yes? I know a moderate amount about the physics and biochemistry of the retina, from which I can hazard some guesses...
You know that "green" can come from "reflects only green" OR equally well from "reflects *everything but* violet." So, the sets of cones in the retina are getting quite different signals from those two different situations: In the first, the green cones are all lit up, while the red and blue cones are pretty quiet. In the second, the blue and green cones are stimulated while the red cones are less so (but maybe not none). So, I'm guessing that at some point these two signal patterns get collapsed to a single interpretation: "green." I.e., we don't think in CYMK notation; the three cone sets and the rods all get combined at some point, and it's this combined and interpreted data that Area V4 is processing.
That's a total SWAG, but perhaps it's useful? |