The Fungible Audio-Visual Mapping and its Experience

Adriana Sa, Baptiste Caramiaux, Atau Tanaka



This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations between the sounds themselves. The question is, how can an audio-visual mapping produce a sense of causation, and simultaneously confound the actual cause-effect relationships. We call this a fungible audio-visual mapping. Our aim here is to glean its constitution and aspect. We will report a study, which draws upon methods from experimental psychology to inform audio-visual instrument design and composition. The participants are shown several audio-visual mapping prototypes, after which we pose quantitative and qualitative questions regarding their sense of causation, and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components – sufficient complexity to be confusing. As the specific cause-effect concepts remain inconclusive, the sense of causation embraces the whole. 


Audio-visual Mapping; Perception; Causation; Multisensory Integration; User Studies.

Full Text:



Bergson, Henri L. Matter and Memory. London: Allen, 1910.

Bregman, Albert S. Auditory Scene Analysis: The Perceptual Organization of Sound. Cambridge: MIT Press, 1990.

Calvert, Gemma A., Charles Spence and Barry E. Stein, eds. The Handbook of Multisensory Processes. Cambridge: MIT Press, 2004.

Chion, Michel. Audio-Vision: Sound on Screen. New York: Columbia University Press, 1994.

Kant, Immanuel. Kritik der Urteilskraft (1790), ed. H.F. Klemme. Hamburg: Meiner, 2001.

Knudsen, Eric I. "Fundamental Components of Attention." Annual Review of Neuroscience 30, no. 1. (2007): 57-78.

Kobayashi, M., Y. Osada and M. Kashino. "The Effect of a Flashing Visual Stimulus on the Auditory Continuity Illusion." Perception & Psychophysics 69, no. 3 (2007): 393-399.

Koffka, Kurt. Principles of Gestalt Psychology. London: Lund Humphries, 1935.

Kubovy, Michael and Michael Schutz. "Audiovisual Objects." Review of Philosophy and Psychology 1 (2010):41-46.

McGurk, Harry and John MacDonald. "Hearing Lips and Seeing Voices." Nature 264 (1976): 746–748.

Palmer, Steve. Vision Science, Photons to Phenomenology. Cambridge, MA: MIT Press, 1999.

Pick, Herbert L., David H. Warren and John C. Hay. "Sensory Conflict in Judgments of Spatial Direction." Perception and Psychophysics 6, no. 4 (1969): 203–205.

Pressing, Jeff. "Some Perspectives on Performed Sound and Music in Virtual Environments." Presence 6, no. 4 (1997): 482-503.

Rubin, Edgar. Visuell Wahrgenommene Figuren: Studien in Psychologischer Analyse. Copenhagen: Gyldendalske Boghandel, 1921.

Sa, Adriana. "How an Audio-Visual Instrument Can Foster the Sonic Experience." Live Visuals, eds. L. Aceti, S. Gibson, S. M. Arisona, O. Sahin, Leonardo Almanac 19, No. 3, (January 2013): 284-305.

Sa, Adriana. "Repurposing Video Game Software for Musical Expression: a perceptual approach." Proceedings of NIME - Conference on New Interfaces for Musical Expression (2014).

Sa, Adriana. "Exploring Disparities Between Acoustic and Digital Outputs". In Leonardo Transactions, ed. A. McLean (2014)

Schaeffer, Pierre. Traité des Objets Musicaux: Essai Inter-Disciplines. Paris: Éditions du Seuil, 1966

Schutz, Michael and Michael Kubovy. "Causality and Cross-Modal Integration." Journal of Experimental Psychology: Human Perception and Performance 35, no. 6 (2009): 1791– 1810.

Shams, L., Y. Kamitani and S. Shimojo. "Visual Illusion Induced by Sound." Cognitive Brain Research 14, no. 1 (2002): 147-152. < a href="">

Sinnett, Scott, Charles Spence and Salvador Soto-Faraco. "Visual Dominance and Attention: The Colavita Effect Revisited." Attention, Perception & Psychophysics 69, no. 5 (2007): 673-686

Snyder, Bob. Music and Memory: An Introduction. Cambridge: MIT Press, 2001.

Stevens, Meghan. Music and Image in Concert. Music and Media, 2009.

Wertheimer, Max. "Laws of Organization in Perceptual Forms." A Source Book of Gestalt Psychology, ed. W. Ellis, 71-88. London: Routledge & Kegan Paul, 1938.



  • There are currently no refbacks.

Journal of Science and Technology of the Arts
Revista de Ciência e Tecnologia das Artes
ISSN: 1646-9798
e-ISSN: 2183-0088
Portuguese Catholic University | Porto

Esta revista científica é financiada por Fundos Nacionais através da FCT – Fundação para a Ciência e a Tecnologia

 Governo da República Portuguesa