The Human Visual System: Digital Cinema

The human visual system is a lot more complicated than we might imagine. A recent paper published in the journal Science Advances (January 12, 2022), Illusion of visual stability through active perceptual serial dependence, by researchers Mauro Manassi (University of Aberdeen, UK) and David Whitney (UC Berkeley), takes this idea a step further. I can't pretend to have slogged through the bulk of this article. The text is dense with the sort of science-speak common to experts in their field of expertise but nearly incomprehensible to the layman (a worldwide issue over the past two years, but I digress!)

The paper begins with the widely accepted theory that our visual system, roughly speaking, is digital. What we see in any instant is an average of a series of these stationary digital images extending back several seconds, enabling the images we perceive to be smooth rather than a jittery string of stationary pictures. In that respect, it's much like the frames in a motion picture.

To quote from the article, "Why do objects in the world appear to be so stable despite constant changes in their retinal images? Retinal images continuously fluctuate because of many sources of internal and external noise ranging from retinal image motion, occlusions and discontinuities, lighting changes, and perspective changes, among many other sources of noise. However, the objects do not appear to jitter, fluctuate, or change identity from moment to moment. This problem—why the world seems unchanging over time—is decades, if not centuries, old."

To quote further, "Despite a noisy and ever-changing visual world, our perceptual experience seems remarkably stable over time. How does our visual system achieve this apparent stability? Here, we [the authors] introduce a previously unknown visual illusion that shows direct evidence for an online mechanism continuously smoothing our percepts over time. As a result, a continuously seen physically changing object can be misperceived as unchanging. We find that online object appearance is captured by past visual experience up to 15 seconds ago. We propose that, because of an underlying active mechanism of serial dependence, the representation of the object is continuously merged over time, and the consequence is an illusory stability in which object appearance is biased toward the past [emphasis mine]. Our results provide a direct demonstration of the link between serial dependence in visual representations and perceived visual stability in everyday life."

The bottom line here can be summarized in the possibility that what we see is a smoothed representation of the world, biased toward what it was up to 15 minutes ago. The implications here would be worthy of a Philip K. Dick novel were he still with us. (I just re-watched large chunks of The Man in the High Castle, an excellent Amazon Prime series undone by a rushed and final 4th season. Why can't TV writers figure out how to end things—looking at you, Game of Thrones!).

If I'm reading the article correctly (and, again, I soon gave up on some of the heavy hitting in the text), what it appears to say is that what we perceive is an average of the stationary eye images from up to 15 seconds ago, but biased toward the earlier "frames." The experiments performed by the Manassi and Whitney appear to confirm that conclusion. That might explain otherwise often inexplainable accidents, including plane and car crashes. We're living in a world that often moves many times faster than it did when our perception systems developed!

That's not to say that we're actually living our lives 15 seconds in the past! It awould appear intuitively obvious that these conclusions might apply only when viewing relatively stable objects. The visual system isn't averaging totally different "frames." Perhaps once you move your head, or something appears that radically changes what's in your visual field, the cycle resets before the average reaches the postulated 15 seconds.

There might be some interesting applications of this knowledge to video display technology. I can't imagine precisely what they might be, but the findings here could be worth exploring by video engineers and designers.

The paper covers only the human visual system, but what about our other senses, particularly hearing? The human hearing system appears to be analog. And since our senses must synchronize (otherwise our observed world would have lip sync issues, among other oddities!), if the authors' thesis is correct the brain must somehow align all of these sensual inputs. More grist for the research mill, but for now my head hurts!

Traveler's picture

The homo sapien brain isn't digital. This is why the smartest "ai" is so hopeless in the real world, among other things.

Thomas J. Norton's picture
I can't point to any personal research suggesting that the human visual system and brain operates digitally in the same sense as our digital technology. But the article in question does refer to research (if I'm understanding it correctly) suggesting that the human visual system operates sequentially, retaining a series of static visual, samples and smoothing them in the brain so that we perceive a smoothed world. I can't argue this either way, and can conceive of some potential (if merely subjective) issues with the theory. If you have access to contradictory research you're welcome to point it out here. The brain is an incredibly complex organ that we still don't fully understand--and perhaps never will.