On the Transmutability of Textual Data : Concept and Practices

This paper discusses the creative potential of the transmutability of digital data, while focusing on the exploration of textual material. It begins by addressing the conceptual and creative possibilities associated to the topic, and then discusses artifacts that imply or express transmutability as an artistic concept and method. To this end, we resort to a framework for the description and analysis of these artifacts, focusing on their conceptual dimension, on their mechanics and on the elements of their experience. In particular, we address the concepts they approach through the use of data in textual formats as source information or content, we consider the processes for its manipulation, and describe the resulting sensory manifestations while emphasizing their dynamics and variability. In this manner, this study seeks to highlight how transmutability becomes relevant as an artistic argument, by proposing aesthetic experiences that explore the ubiquity and heterogeneity of data in our contemporary world, as it becomes available in text formats.


| EXPLORING TRANSMUTABILITY
The notion of transmutability, as proposed by Levin (2010), is a principle underlying the creative exploration of digital data, as expressed by the mapping of some input data stream into sounds and graphics.This paper addresses this notion in continuity with a previous study that explored the transmutability of digital data, from its theoretical discussion and manifestations in creative practices, to its practical exploration (Lee et al., 2014).
In order to understand the concept of transmutability we can consider that, within the computer, "all media objects are composed of digital code; they are numerical representations" (Manovich, 2001, p. 27) that, when regarded as raw material, have the potential to be translated into any tangible form through algorithmic manipulation.This creative potential is expressed by practices that rely on software as their medium, and create aesthetic artifacts that involve articulations between the visual and auditory realms, and also other physical or tangible expressions (Lee et al., 2014, p. 417).
As Golan Levin emphasizes, the "premise that any information can be algorithmically sonified or visualized" can be the "starting point for a conceptual transformation and/or aesthetic experience", or a means of "enabling some data stream of interest to be understood, experienced, or made perceptible in a new way" (Levin, 2010, pp. 273-4).In this sense, the notion of transmutability can be associated to other terms that similarly invoke software as a means of exploring digital data as a "self-contained abstraction" in its "inherent malleability" (Whitelaw, 2008).It relates to the transcoding of digital data, as a direct consequence of describing information numerically (Reas et al., 2010, p. 79) and evokes transmediality as a "translatability across media" (Hayles, 2006, p. 194).It can also be associated to transmateriality as a notion that expresses a view of "digital media and computation as material flows (…) transducing anything to anything else" (Whitelaw, 2009).
Considering that any given data can be mapped onto any visual, auditory or tangible form, we have approached the notion of transmutability focusing on visualization and/or sonification, as creative concepts and methods.Accordingly, we have explored their possible articulations, narrowing the scope of our research to sound visualization and image-based sonification processes, given that, although the source data diverges (being respectively sound or image) their logic and representation principles have common traits.Both involve two nodes in the process, "encoding and decoding", meaning, a systematic method of extracting and mapping data relations to sounds or images, as well as the interpretation of relationships contained in that information (Song & Beilharz, 2006, p. 450).
Based on these observations, we developed a practical exploration and illustration of the topic, which we defined as a meta-project [1], since it explored different ways of audio-visually translating the textual content of the study.In this manner, we sought to gradually abstract this textual content, providing a new perception or experience through seeing and hearing.This work resulted from a closed system of correspondences between text, graphic symbols and sound parameters that promote a contemplative experience and, at the same time, highlight the diversity of possible derivations, reinterpretations and subjective approaches to the same referent.This work allowed us to identify the possibilities of using text as source material, pointing towards further explorations of the creative and expressive potential of transmutability.
Therefore, its more recent developments shift the focus from the audiovisual results towards the textual nature of the source data (Lee & Ribas, 2016), through an analytical approach and overview of practices that imply or express transmutability as an artistic concept and method.

TEXT AS SOURCE DATA
The focus on the exploration of textual material relates to the fact that "a lot of the richest information we have" is available in text formats (Viégas as cited in Heer, 2010, p. 7) and "grows on a daily basis" (Nualart-Vilaplana et al., 2014, p. 224).Part of this profusion is associated with the popularity and ease of producing content online, which results in the "availability of large amounts of heterogeneous text data" (Kucher & Kerren, 2015, p. 117).
The interest in exploring the specificities of text as source data, and the potential of finding "visual ways to make them talk" (Viégas as cited in Heer, 2010, p. 2), places text visualization and exploration as a "growing and increasingly important subfield of information visualization" (Kucher & Kerren, 2015, p. 117).
Also, the process of visualizing text is intrinsically associated with text analysis, via "computational linguistics, natural language processing, machine learning and statistics" (Nualart-Vilaplana et al., 2014, p. 224).
In this sense, if we consider the "advances being made in text analysis research" and the "growing amount of accessible data in text format" (Nualart-Vilaplana et al., 2014, p. 224), we can observe how text gains a transformative potential worthy of development and exploration that is tied to the "text processing algorithms" applied to its manipulation and transformation (Kucher & Kerren, 2015, p. 117).
Therefore, we can say that artifacts that explore the inherent mutability of digital data creatively question the "nature of our now ubiquitous data systems", by making data "explicit" and tangible, while probing its "potential, and significance" (Whitelaw, 2008).In this process, different approaches and methods for reconfiguring data may be involved, following mainly analytical or aesthetic purposes.This means that the aim of the project can be to provide a "new reading or understanding of information" or, in turn, to explore digital data in order to "create expressive languages or sensory experiences" (Lee et al., 2014, p. 420).

APPROACHES TO TEXTUAL DATA
In accordance with these analytical or expressive approaches to data, particularly in textual formats, we can distinguish conceptual purposes and aesthetic intents that pertain to an exploration of the formal features of text, to a reflection on the meaning it conveys, or even to the use of text as an abstraction, thus emphasizing the mutability of digital data.Some projects assume textual data per se as the subject matter of the work, that is, they consider text as raw material ("the text as it is") or extract and consider "a representative part of that text" (Nualart-Vilaplana et al., 2014, p. 224) as the result of "text mining algorithms" (Kucher & Kerren, 2015, p. 117).The focus of these works is on the exploration of the formal specificities of text as source material, considering that a text is a one-dimensional structure "organized in a sequential manner", that can have "multiple internal structures", a specific morphology (paragraphs, sentences, words), information structure (chapters, sections), diverse data types or formats (txt, html, etc.) and different patterns, as well as "a subjective component and an abstract structure that is not readily analysed by a computer" (Nualart-Vilaplana et al., 2014, pp. 221-224).
Other projects consider textual data as content that conveys some kind of meaning, or represents a given subject matter.In these cases, the focus is on semantics rather than form, and the aim is to propose a new "understanding, perception or experience" of that content (Levin, 2010, p. 274) or to "portray not merely data, but the personal, emotional reality that the dataset refers to" (Whitelaw, 2008).
Finally, text can be considered as an abstraction, in the sense that what is conceptually emphasized is the translation process itself (Levin, 2010), or the possibility of mapping any kind of data into a new tangible representation.In such cases, and "depending on how the text is treated and processed", it can be detached from its semantics, being that the textual source or origin "is not always relevant" (Nualart-Vilaplana et al., 2014, p. 228).The dataset is treated as "an abstract set of potentials", since "the process doesn't care what the dataset is, or was", and treats it as "just input" (Whitelaw, 2008).What is valued is the expressive and dynamic potential of software as a means of algorithmically transforming any kind of data into a new expressive manifestation.
These different strategies thus expose the potential of translating and revealing inherent, and eventually latent or hidden, dimensions of text in a new expressive form, either relating to its formal specificities, semantic aspects or abstraction.

| TRANSMUTABILITY AS A CREATIVE PRACTICE
In order to provide an overview of the range and scope of creative approaches that are tied to the transmutability of textual data, we selected and analyzed the following set of artworks [2]: 16.Viégas, Fernanda & Martin Wattenberg. History Flow. 200317. Whitelaw, Mitchel. Weather Bracelet. 2009 In order to analyze these works we resorted to the frameworks proposed by Wardrip-Fruin (2006) and Hunicke, LeBlanc and Zubek (2004) for understanding aesthetic artifacts that are digital computational systems, or works that are driven by processes, as dynamic systems [3].These frameworks highlight that, when examining these artifacts, we must consider not only their sensory results or modes of expression but also their procedural modes of expression and dynamics (Ribas, 2014, p. 53).
This analysis emphasizes the alignment between the works' themes and concepts, which are tied to the use of textual data as source information or content, as these are implemented through specific data and algorithmic processes for their manipulation.It also considers the elements of the experience of the work, namely the resulting surface elements and the dynamic behavior that define the works' experience (Lee et al., 2014, p. 423) according to the following dimensions: The conceptual dimension (theme and content) concerns the subject matter of the work, relating to specific approaches to text (its form, meaning or abstraction), while addressing the significance and relevance of transmutability as an artistic argument; When we talk about mechanics (data and processes) we are addressing the implementation of concepts through specific textual data formats and algorithmic processes as constituent elements of the system (e.g.data values and input, as well as mapping processes and their possible articulations); The experience dimension (surface and dynamics) contemplates the sensory outcomes (output formats and modes of expression) and the observable behavior of the work, as aspects pertaining to the nature of the work as a technological and aesthetic artifact, and relating to the variability and determinability of its behavior.Therefore, by considering such views we seek to describe the salient traits of these projects, while tackling the questions that their conceptualization, enactment and experience may raise.

| ANALYSIS 4.1 CONCEPTS: THEMES AND APPROACHES
According to the previously mentioned approaches to textual data, we can identify diverse creative intents and relationships to text as the main referent or subject matter of the work (Figure 1).
In projects that tend to explore the formal and material qualities of text (its format or internal logic), we identify a particular interest in literary works, as "a field that, apart from being characterized by complex combinations of words, can present high levels of human abstraction and freedom of structure and experimentation" (Nualart-Vilaplana et al., 2014, p. 234).Works such as Ben Fry's On the Origin of Species: The Preservation of Favoured Traces (2009) give us a perception of the evolution of scientific ideas and the gradual refinement of Darwin's discourse over several editions of the book (Figure 2).Another case is History Flow (Viégas & Wattenberg, 2003) that visualizes and reveals patterns emerging from the editing history of Wikipedia articles.
Other projects, in turn, focus on content, using text as a means to explore a given subject or theme.Rather than focusing on the text format, these projects focus on the meaning that the text conveys, seeking to express or portray the reality that the textual data refers to, as an "index of reality" (Whitelaw, 2008).For example, We Feel Fine (Harris & Kamvar, 2006) is defined as an "exploration of human emotion", by gathering "emotional data" on a global scale, through the search of blog entries with occurrences of the phrases "I feel" and "I am feeling".Similarly, Listening Post (Rubin & Hansen, 2001) provides an audiovisual reading of online conversations in real-time, by collecting data from unrestricted blogs and forums, as a reflection on the "immediacy of virtual communication" (Figure 3).
Additionally, other projects use textual data as an abstraction, that is, as raw material, or as some kind of textual codification that can be used as input, regardless of its source or meaning.What these projects put to the fore is the malleability of text as digital data, and thus the computational processes applied to its manipulation, or the possibility of translating "anything" into "anything else" (Whitelaw, 2008).This is the case in Spam Architecture (Dragulescu, 2005) where patterns, keywords and rhythms found in junk mail are processed and translated into three-dimensional models allusive to architectural forms (Figure 4).Another example is ZNC Browser 2.0 (Luining, 2003) that seeks to reveal the "arbitrariness of code" as a "conceptual piece" that automatically translates the HTML code of web pages  into a sequence of sounds and colors, thus proposing an abstract "sonic browser".

MECHANICS: DATA AND MAPPING PROCESSES
When we look at these systems from the point of view of their mechanics, we can distinguish different forms of data collection, kinds of input (and their values), as well as different visualization and sonification methods or mapping processes (Figure 5).
Many of the projects analyzed rely on a fixed dataset as input that is inserted into the system by its author.This dataset is then explored as a whole, allowing the development of visual and/or auditory expressions that seek to reveal the complexity and inherent structure of the data, namely when spatially or temporally displayed.An example of that is Shakespeare Machine (Rubin & Hansen, 2012) that pulls out "interesting speech patterns" that emerge from every Shakespeare's play (Figure 6).However, other projects use a continuous data stream, whose values are changing in real time, or chunks of information that gradually update the values.These streams or chunks are usually captured through computational processes and inserted into the system automatically, as in 1:1 (Jevbratt, 1999(Jevbratt, -2002) ) that uses web crawlers to search for IP addresses, which are then stored in databases that are visualized through different interfaces (Figure 7).
In terms of mapping processes, we observed that most of these works employ visualization methods, proposing a purely visual expression of data, while  few projects make use of sound or sonification in addition to (or as a complement to) the visualization procedures.That is the case with Hard Data (DuBois, 2009), in which the author seeks to re-contextualize "formal stochastic music in the context of real-world statistics", while creating abstract audiovisual experiences based on data from the American military actions in Iraq.Another example is Listen to Wikipedia (Hatnote, 2014), which presents us with a "real-time auralization of Wikipedia growing", based on feeds of the latest changes, represented by simple sounds and coloured circles (Figure 8).Although fewer projects make use of sonification processes, the use of sound as a mode of expression is also explored in The Sound of Ebay (Ubermorgen, 2008), as it generates "unique songs by using ebay user-data".

SURFACE: SENSORY RESULTS AND EXPRESSION
The projects mentioned above already suggest the diversity of sensory modes of expression and formal aspects of representation we can observe in these projects, and which are tied to their specific approaches to textual data as subject matter (Figure 9).
While most of the projects analyzed entail mapping processes that result mostly in visual or auditory expressions, we also included works that propose a physical and material rendering of data.In Weather Bracelet (Whitelaw, 2009) the author creates a "wearable data-object" generated from daily weather data sourced from the Bureau of Meteorology (Figure 10).In turn, News Knitter (Kurbak & Yavuz, 2007-2008) produces knitted garments as "an alternative medium to visualize large scale data" gathered from daily political news.
However, in terms of formal aspects of representation and expression it is not always evident what aspects or parts of the text are actually represented in the output, be it through visualization or sonification processes.While some projects present the whole text in the output, as On the Origin of Species: The Preservation of Favoured Traces (Fry, 2009), projects  such as Shakespeare Machine (Rubin & Hansen, 2012) present selected parts of the source material as fragments of speech that "appear, dissolve, and move like a choreographic dance", according to an algorithm that sets rules for the combinations of words.In turn, Bible Cross-References (Harrison, 2008) doesn't present textual information as output, as it privileges "something more beautiful than functional".
Nevertheless, most of the projects analyzed approach structural aspects of the text, such as grammatical or morphological attributes, as parameters that are mapped into graphic or audio features.Many of these works resort to abstract elementary figures and sounds that, when combined, can reveal unexpected patterns or rhythms, or even complex configurations emerging from the data.For instance, Pure Data Read as Pure Data (Maigret, 2010) translates the source code of the application Pure Data into sounds and colored pixels, in order to promote a "physical experience of the digital data" (Figure 11).

BEHAVIOR: DYNAMICS AND VARIABILITY
Adding to the mentioned formal aspects of expression, the source of data also influences the nature of the output and the dynamic behavior of the work, depending on whether the work is open or not to interaction with external input (Figure 9).
The use of a fixed dataset usually corresponds to a system that is closed to external input.As such, the output is an instance that the system generates each time it runs, resulting in either a static or a transient (non-variable) output, thus promoting a contemplative experience based on the formal or semantic qualities of the source data.The output can be a static image resulting from a process of 'filtering', such as a selective "snapshot" of the final state of the work or of "accretions" of processes over time (Dorin et al., 2012, p. 247).For example, the project Bible Cross-References (Harrison, 2008) presents a global view of the "textual cross references found in the Bible" through diagrams that "honor and reveal the complexity of the data" (Figure 12).
When the output is transient, as a time-based or animated sequence (usually in response to a time dependent dataset), the work privileges a perception of patterns emerging from the text or a way to "understand or follow its evolution over time" (Nualart-Vilaplana et al., 2014, p. 230).This can be observed in On the Origin of Species: The Preservation of Favoured Traces (Fry, 2009), where animated visualizations demonstrate the changes and additions of text over the successive editions of the book.
Conversely, a continuous data stream can be used to gradually determine output variations, providing an immediate perception of input fluctuations coming from external data sources or processes.That is the case in Listening Post (Rubin & Hansen, 2001) that culls information from online sources in real time.Also, in We Feel Fine (Harris & Kamvar, 2006) we can see that the interface changes and evolves as new updates in the blog entries are found (Figure 13).
In addition, a significant number of projects are presented as online interfaces, allowing the user to explore or navigate different views.Therefore, the experience of the output becomes varied, even if the system is not necessarily producing variable results while acting on the same input.For instance, in 1:1 (Jevbratt, 1999(Jevbratt, -2002) ) the user is allowed to navigate through the interface, being able to "query the (visualization) system and obtain a unique representation for each search" (Nualart-Vilaplana et al., 2014, p. 230).Similarly, in The Dumpster (Levin et al., 2006) the user "can surf through tens of thousands of specific romantic relationships".Among these projects we can also observe a different kind of interaction, namely with Type/Dynamics (LUST, 2013), in which the audience is not actively or intentionally manipulating the source data but "sensors track the visitors' movement" causing the output to change as it reacts to "the position and number of visitors in the space, as well as their distance from the gallery walls" (Figure 14).

| DISCUSSION
According to these observations we can highlight what these projects share as a creative exploration of textual data, and how they diverge, regarding their specific approaches to the source material, as well as the different aesthetic intents and kinds of experience they propose.
According to this selection of projects, we can see that projects that use literary works as source material put an emphasis on formal aspects of the texts, as they present a "high level of abstraction and little formal structure" (Nualart-Vilaplana et al., 2014, p. 228).The potential lack of regularity in terms of vocabulary, or length of the texts, and their subjective discourse structure, results in more creative freedom and expressive possibilities because there are no given conventions or rules for representation.
The fixed nature of literary texts is usually associated with a sequential analysis of the whole, that is, the visualization often follows the texts' sequence or order.One exception is Shakespeare Machine (Rubin & Hansen, 2012) in which parts of the text are selected according to different rules, and the reference to the original text sequence is then discarded.
When the focus is on meaning, a wider scope of themes emerges, ranging from human social dynamics (e.g.virtual online communication, identity, or different kinds of statistics), to natural phenomena (e.g.meteorological data), or even to the density and complexity of the web and its navigational structure.These projects tend to work either with a fixed dataset or with sequential updates of that data, presenting an indexical narrative of a reality, hence putting to the fore its latent, or even hidden, patterns.
This is an aspect that is reflected in different kinds of sensory results or modes of expression (audio, visual or other), being that, in terms of format, these works range from audiovisual installations to online  interfaces, as well as material and physical renderings of the source data.
Finally, the exploration of text as an abstraction is mostly related to an analysis of data as raw material, pertaining, for example, to web content or digital data that can be readily analyzed through computational means -it can be considered as it is, and subjected to any kind of arbitrary mapping, therefore emphasizing its "malleability" and "susceptibility to transformation" (Whitelaw, 2008).The dataset is detached from any given or a priori meaning and treated according to a subjective process or conceptual approach, so it doesn't suggest a relationship to the source text.Since the nature of the source data does not determine or condition the mapping process, this kind of approach is more prone to involve sonification and audiovisual results.

| CONCLUSION AND FUTURE WORK
The previous discussion also suggests that the aesthetic experience of these works is not merely focused on their sensory results, but on the understanding of the processes leading to the observable results.Accordingly, we can consider that "what we experience, even as static displays", are the results of "software performances", which give us not objects but instances or occasions for experience (Manovich, 2013, p. 33).So we can interpret these outputs as the products of processes.In this sense, these projects entail a process of "procedural interpretation" or an understanding of the work that often involves "mental simulations of the processes behind the surface" (Carvalhais & Cardoso, 2015, pp. 143-144).
According to this idea, we acknowledge the potential for a deeper examination of these forms of procedural interpretation, namely, through a refinement of the framework concerning the distinctive features of the experience of these artworks.This implies an analysis of their dynamics and variability, and therefore, further discussion on what we consider to be the aesthetic artifact in question, the system and/or the outcomes it presents to the audience as instances or events.Consequently, when examining the dynamics of the work, it is important to consider both the variability of the software system and the variability of the outputs it produces, given that the aesthetic artifact can be considered both, as in 1:1 (Jevbratt, 1999(Jevbratt, -2002)).In this sense, when we look at online interfaces, even if the system is closed to interaction with external data or audience input, the audience may explore distinct views or results shown in a different order or appearance.
Among the projects analyzed, few propose an exploration of textual data through sonification.However, since sound is a linear and time-based medium that exists in "time over space", and has "abstract attributes (…) such as pitch, loudness and timbre" that are "presented simultaneously" and are "subjective", sonification can be an efficient alternative or complement to visualization, and eventually present new insights on the data that are more related to a "symbolic meaning" (Song & Beilharz, 2006, pp. 450-451).This is something we consider worthy of exploration, also given the high level of abstraction and formal structure of texts, thus their openness to subjective interpretation.
Furthermore, an examination of a broader scope of systems that are open to interaction with external data or processes can be of interest, in particular, considering human input, or audience interactive work as well as the possibilities that are given to the audience for accessing, influencing or determining variable outcomes.
Acknowledging the multiplicity of transmutability as a creative concept and practice, this study sought a deeper understanding of artistic approaches to textual data, highlighting their focus on form, content or abstraction.To this end, it described a set of aesthetic artifacts according to a framework focused on their themes and subject matter (concepts), their data and processes (mechanics) and their surface and dynamics (as the elements of their experience).With this approach, this study sought to reveal the creative and expressive potential of the mutability of digital data as it becomes accessible in different text formats.It sought to emphasize the relevance of transmutability as an artistic argument that comments on the growing amount of digital data that permeates our contemporary world.
ENDNOTES [1] The project 01.t02.d03.a has been included in the xCoAx 2014 exhibition in Porto, and its online documentation can be accessed in: http://catarinalee.fbaul-dcnm.pt/.[2] We assume a correspondence to the topic when the artifacts present the following criteria: (1) use software as medium; (2) explicitly work on or explore information in textual format; (3) entail visualization and/or sonification methods; (4) and their results emphasize the significance of data and/or the transformational process involved as subject matter of the work.
[3] The model proposed by Wardrip-Fruin addresses the interplay between data, processes, surface, interaction, author and audience (2006, pp. 9-11).It also considers the "forms and roles" of computation that distinguish the ways in which the work operates, according to its computational variability, interaction and source of interaction (2006, p. 398).In addition, the MDA framework provides different but interrelated perspectives focused on their mechanics, dynamics and aesthetics (Hunicke et al., 2004).

Figure 1 |
Figure 1 | Analysis graph: conceptual dimension -observation of artwork's theme and content.

Figure 5 |
Figure 5 | Analysis graph: mechanics dimension -observation of artwork's data and mapping processes.

Figure 9 |
Figure 9 | Analysis graph: experience dimension -observation of artwork's surface expression and system behavior.