The Importance of Synesthesia in Artificial Intelligence

Through the early history of computing, information flowed in through digital text and out through digital text. Artificial intelligence was thought by many to be science fiction or imagination. Now, AI has exploded, with AI engagements, such as chat bots, voice assistants, autonomous vehicles, medical diagnostics, robotics and human-AI relationships, increasingly interwoven into our everyday lives.

In years ahead, AI will become even more powerful, more pervasive, more personal, and much more ethically significant. The future of AI is beyond technology—it is increasingly about our choices and requires our attentive understanding. The AI applications we are building reflect the values we embed. It is up to us to make the AI of the future intelligent, wise, ethical and worthy.

To understand the full potential of AI, one must understand the social media implications of synesthesia. Synesthesia is a phenomenon through which sensory crossovers occur, causing individuals to experience one sense as affected by another. Our senses of sight, hearing, touch, taste, and smell singularly and in combination influence our interactions with others and with our environment. (Synesthesia comes from the Greek synth meaning together, and esthesia, meaning perception.) Every person experiences his or her individual level of synesthesia. (Cytowic 2024)

For AI to fully simulate human behavior, one must experience the blending of the senses and their effects on understanding and human behavior. The film Avatar, for example, used AI to create characters as perfectly human as possible. The “uncanny valley”—a term first introduced in 1970— refers to a foundational theory in media psychology, recognized by the film’s director, James Cameron, that characters developed by AI will elicit an eerie feeling as they approach resemblance to human behavior and then elicit full empathy once they reach perfect resemblance. A box office mega-success, Avatar is lauded as fueling a paradigm shift in avatar development.

Human experience is powerfully influenced by language. Accurate communication requires understanding of the implications of blending semantics, semiotics, and their combined effects. Semantics explains words and meaning. Semiotics is the study of signs and symbols, including iconography. Integrating the senses is a major challenge in the creation of artificial intelligence, and it is amplified in robotics. This raises questions such as, how close to actual human behavior can robotics become with an avatar?

We now know that intelligence is not limited to words. It is becoming clearer that the AI of the future will be multidimensional and multisensory. We are just beginning to understand computer applications across modalities, and the implications of AI and the human senses in years ahead of us. Thus, the importance of understanding synesthesia, and the blending of the senses in thought, feelings and behavior.

Computer power in AI is important because it enables significant change. We are now on the threshold of 6G in common use. In the decades ahead, 6G, 100 times more powerful than 5G, is expected to revolutionize the ways we communicate. It will enable applications to engage in real-time holography and higher levels of simulated reality in sensory communications.

My view is that augmented reality and virtual reality will advance to levels never experienced, and 6G will enable more effective blending of sensory emulation in all forms of AI. With 6G still several years away, visionaries are already predicting many types of advances. And, though 7G is still distant, it holds even more significant potential in the future.

Researchers in the American Psychological Association’s Society for Media Psychology and Technology are among the many now studying AI and human behavior. The AI of the future will increasingly emulate human behavior in all of its manifestations.

As computer power advances, we must match it with understanding of the profound psychological nature of AI in human relationships. More study is needed and the opportunity, timing, and sharing of findings are social imperatives. The blending of the sense, one way or another through AI is a game changer, and the technology to increase its influence is on our horizon and coming fast.