The Fungible Audio-Visual Mapping and its Experience

Adriana Sa; Baptiste Caramiaux; Atau Tanaka

doi:10.7559/citarj.v6i1.131

Adriana Sa EAVI group, Department of Computing, Goldsmiths University of London | Lisbon | Portugal
Baptiste Caramiaux EAVI group, Department of Computing, Goldsmiths University of London | London | UK
Atau Tanaka EAVI group, Department of Computing, Goldsmiths University of London | London | UK

DOI: https://doi.org/10.7559/citarj.v6i1.131

Keywords: Audio-visual Mapping, Perception, Causation, Multisensory Integration, User Studies.

Abstract

This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations between the sounds themselves. The question is, how can an audio-visual mapping produce a sense of causation, and simultaneously confound the actual cause-effect relationships. We call this a fungible audio-visual mapping. Our aim here is to glean its constitution and aspect. We will report a study, which draws upon methods from experimental psychology to inform audio-visual instrument design and composition. The participants are shown several audio-visual mapping prototypes, after which we pose quantitative and qualitative questions regarding their sense of causation, and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components – sufficient complexity to be confusing. As the specific cause-effect concepts remain inconclusive, the sense of causation embraces the whole.

Author Biographies

Adriana Sa, EAVI group, Department of Computing, Goldsmiths University of London | Lisbon | Portugal

Adriana Sá is transdisciplinary artist, musician, performer/composer. Between 1995 and 2009 she worked fulltime on her artistic projects, involving many international artists, with grants and commissions. In 1997 she started using sensor technologies to explore music connected to light, movement, architecture and weather. Currently she explores disparities between human perception and digital analysis as creative material. She has been clarifying artistic insights with the aid of cognition/attention research, and developing 3D audio-visual software that operates based on pitch detection from a custom zither input. Adriana performed and exhibited worldwide, e.g. at Calouste Gulbenkian and Serralves (Portugal), Experimental Intermedia and PS1/MoMa (US), Caixa Forum and Arteleku (Spain), ICA–Institute of Contemporary Arts (UK) or Aomori Contemporary Art Center (Japan). Since 2011 she is a researcher in Arts and Computing Technologies at Goldsmiths, with international, peer-reviewed publications in Leonardo, Leonardo Transactions, Live Interfaces Proceedings, xCoAx Proceedings and NIME Proceedings.

Baptiste Caramiaux, EAVI group, Department of Computing, Goldsmiths University of London | London | UK

Baptiste Caramiaux is a researcher in the EAVI group, at Goldsmiths, University of London, since 2012. His work focuses on Human Motion, Interaction Design, Computational Systems and Sound Perception. He received a PhD in acoustic, signal processing and computer science applied to music from University Pierre et Marie Curie (UPMC), in December 2011, completed at IRCAM – Centre Pompidou. Before joining IRCAM, he was trained in applied mathematics and music. His work has been published in international peer-reviewed journals (ACM Transactions, Computer Music Journal, etc.), conferences (NIME, CHI, SMC, etc.) and festivals (Transmediale, etc.).

Atau Tanaka, EAVI group, Department of Computing, Goldsmiths University of London | London | UK

Atau Tanaka creates musical instruments using sensing technology to capture movements and gestures of musicians. His first inspirations came upon meeting John Cage during his Norton Lectures at Harvard. Atau then studied at CCRMA Stanford, and conducted research in Paris at IRCAM, Centre Pompidou. He has been artistic ambassador for Apple, was the first artist to be engaged as researcher at Sony Computer Science Laboratory (CSL), and has been mentor at NESTA UK, and Artistic Co-Director of STEIM Amsterdam. His work is funded by the European Research Council (ERC). He is Professor of Media Computing at Goldsmiths.

References

Bergson, Henri L. Matter and Memory. London: Allen, 1910.

Bregman, Albert S. Auditory Scene Analysis: The Perceptual Organization of Sound. Cambridge: MIT Press, 1990.

Calvert, Gemma A., Charles Spence and Barry E. Stein, eds. The Handbook of Multisensory Processes. Cambridge: MIT Press, 2004.

Chion, Michel. Audio-Vision: Sound on Screen. New York: Columbia University Press, 1994.

Kant, Immanuel. Kritik der Urteilskraft (1790), ed. H.F. Klemme. Hamburg: Meiner, 2001.

Knudsen, Eric I. "Fundamental Components of Attention." Annual Review of Neuroscience 30, no. 1. (2007): 57-78. http://dx.doi.org/10.1146/annurev.neuro.30.051606.094256

Kobayashi, M., Y. Osada and M. Kashino. "The Effect of a Flashing Visual Stimulus on the Auditory Continuity Illusion." Perception & Psychophysics 69, no. 3 (2007): 393-399. http://dx.doi.org/10.3758/BF03193760

Koffka, Kurt. Principles of Gestalt Psychology. London: Lund Humphries, 1935.

Kubovy, Michael and Michael Schutz. "Audiovisual Objects." Review of Philosophy and Psychology 1 (2010):41-46. http://dx.doi.org/10.1007/s13164-009-0004-5

McGurk, Harry and John MacDonald. "Hearing Lips and Seeing Voices." Nature 264 (1976): 746–748. http://dx.doi.org/10.1038/264746a0

Palmer, Steve. Vision Science, Photons to Phenomenology. Cambridge, MA: MIT Press, 1999.

Pick, Herbert L., David H. Warren and John C. Hay. "Sensory Conflict in Judgments of Spatial Direction." Perception and Psychophysics 6, no. 4 (1969): 203–205. http://dx.doi.org/10.3758/BF03207017

Pressing, Jeff. "Some Perspectives on Performed Sound and Music in Virtual Environments." Presence 6, no. 4 (1997): 482-503.

Rubin, Edgar. Visuell Wahrgenommene Figuren: Studien in Psychologischer Analyse. Copenhagen: Gyldendalske Boghandel, 1921.

Sa, Adriana. "How an Audio-Visual Instrument Can Foster the Sonic Experience." Live Visuals, eds. L. Aceti, S. Gibson, S. M. Arisona, O. Sahin, Leonardo Almanac 19, No. 3, (January 2013): 284-305.

Sa, Adriana. "Repurposing Video Game Software for Musical Expression: a perceptual approach." Proceedings of NIME - Conference on New Interfaces for Musical Expression (2014).

Sa, Adriana. "Exploring Disparities Between Acoustic and Digital Outputs". In Leonardo Transactions, ed. A. McLean (2014)

Schaeffer, Pierre. Traité des Objets Musicaux: Essai Inter-Disciplines. Paris: Éditions du Seuil, 1966

Schutz, Michael and Michael Kubovy. "Causality and Cross-Modal Integration." Journal of Experimental Psychology: Human Perception and Performance 35, no. 6 (2009): 1791– 1810. http://dx.doi.org/10.1037/a0016455

Shams, L., Y. Kamitani and S. Shimojo. "Visual Illusion Induced by Sound." Cognitive Brain Research 14, no. 1 (2002): 147-152. < a href="http://dx.doi.org/10.1016/S0926-6410(02)00069-1">http://dx.doi.org/10.1016/S0926-6410(02)00069-1">http://dx.doi.org/10.1016/S0926-6410(02)00069-1>

Sinnett, Scott, Charles Spence and Salvador Soto-Faraco. "Visual Dominance and Attention: The Colavita Effect Revisited." Attention, Perception & Psychophysics 69, no. 5 (2007): 673-686 http://dx.doi.org/10.3758/BF03193770

Snyder, Bob. Music and Memory: An Introduction. Cambridge: MIT Press, 2001.

Stevens, Meghan. Music and Image in Concert. Music and Media, 2009.

Wertheimer, Max. "Laws of Organization in Perceptual Forms." A Source Book of Gestalt Psychology, ed. W. Ellis, 71-88. London: Routledge & Kegan Paul, 1938. http://dx.doi.org/10.1037/11496-005