Agency and Algorithms

Although the concept of algorithms has been established a long time ago, their current topicality indicates a shift in the discourse. Classical definitions based on logic seem to be inadequate to describe their aesthetic capabilities. New approaches stress their involvement in material practices as well as their incompleteness. Algorithmic aesthetics can no longer be tied to the static analysis of programs, but must take into account the dynamic and experimental nature of coding practices. It is suggested that the aesthetic objects thus produced articulate something that could be called algorithmicity or the space of algorithmic agency. This is the space or the medium – following Luhmann’s form/medium distinction – where human and machine undergo mutual incursions. In the resulting coupled “extimate” writing process, human initiative and algorithmic speculation cannot be clearly divided out any longer. An observation is attempted of defining aspects of such a medium by drawing a trajectory across a number of sound pieces. The operation of exchange between form and medium I call reconfiguration and it is indicated by this trajectory.


| INTRODUCTION
Algorithms are old hat.The term originates from roughly a millennium ago, and the modern use began in the first half of the last century (Blass & Gurevich, 2003), just before the advent of the digital computer.It is thus interesting to ask why in the past few years there has been a strong "renaissance" in the discourse on algorithms, in society, in science and technology, and in the arts.Is it simply because computation has become so inexpensive that algorithms pervade our daily lives?Is it simply that electronic art has said everything since the 1970s, and we are just in a period of repetition that is concealed by the increased throughput of computers, the increased complexity and high resolution of algorithms that run in real-time?
What seems to have changed in the past decades is the philosophical and conceptual underpinning concerning the relationship between humans and machines, and especially the question of agency when the two go together.After the different waves of cybernetics, after information and control theory, semiotics and linguistics, cognitive science and artificial intelligence, we are reaching a point that, depending on the school of thought, could be characterised either by an intensified constructivism, or by a renewed realism, both of which de-emphasise the human subject and the categorial split between humans and machines.From this standpoint, the interesting question is not so much whether machines can be creative or artistic, but rather how the exchange and assimilation processes between human and machine are structured, and how they can give rise to an aesthetics (or epistemology) founded in such compound, "mereotopological" agency, i.e. one where the relations between the individuals and the whole, and where the formation of space are crucial.

| ORIGINS OF ALGORITHMIC AGENCY
Before shedding light on this strange agency, perhaps a few things need clarification.First of all, one may ask what exactly an algorithm is, and what its relationship to machines is.One might think it should be welldefined in its proper subject area, computer science.And indeed there are very succinct definitions such as the one given by Kowalski (1979): "Algorithm = Logic + Control".The logic part is comprised of definitions for abstract procedures related to the knowledge about the problem domain, and of data structures on which these procedures operate, while the control part is concerned with strategies for turning the logic component into an efficient machine, strategies for unwinding the knowledge in time and space.This is illustrated in Figure 1.
Two things are apparent from the figure: Here, algorithms are thought to be separate from their environment, they can be taken and re-applied elsewhere without further ado.Also, algorithms, although they may process temporal data and although they need time to process data, appear as static structures that neither have a history of coming into existence, nor any providence of future transformation.In other words, in the tightness of language and cybernetics, there is no space left for performativity that goes deeper than an abstract analysis of space/time requirements.
The concept of algorithm in computer science is far from undisputed.For example, a brief discussion by Gurevich (2012) poses the question: "Can the notion of algorithm be rigorously defined?" and then answers with yes and no.On the one hand, specific notions of algorithms have become stable and "crystallised", on the other hand the notion is ever expanding with new kinds of algorithms.An adumbration of algorithms in terms of computer science would usually state that they are something abstract, that they operate on objects and "compute" them.Some feel that one should not confuse an abstract algorithm with a concrete implementation, others withdraw from the position that an ontology of algorithm is actually useful and that it suffices to describe its functional properties, etc. Gurevich hints at the limits of the usefulness of abstract definitions by stating that each algorithm has a "natural level" of abstraction which may vary, and furthermore that a purely declarative concept of algorithms overlooks that "… every piece of software is an algorithm … As software is developed, it evolves.A book with a declarative specification quickly becomes obsolete.If specification is not executable, you cannot experiment with it."(Gurevich, 2012, p. 40) The agency I want to talk about is precisely the one linked to the possibility of engaging with algorithms, to experiment with them.But if we want to direct the attention to the aesthetic consequences, it is also advisable to free oneself from a perhaps too narrow view of what algorithms are.There is nothing wrong with allowing a spectrum of meanings, with acknowledging that a "machine" can be an abstract mathematical concept like the Turing machine, but it can also be a vehicle of thought, as it happens when Deleuze and Guattari (1983, p. 36) excessively formulate *-machines, being generally understood as systems of interrupting flows, in which the interruptions or cuts paradoxically ensure the continuity of the flow that is associated with another machine; and so a machine is always connected to yet another machine ad infinitum.In relation to algorithms, Parisi's (2013) analysis points to a similar direction: Instead of generative aesthetics based on prediction and probabilities, she argues that there is a speculative tendency intrinsic to computation, producing genuine novelty that cannot be explained by external forces or initial conditions.
The technical definitions of algorithms may look appealing due to their conciseness and apparent precision, compared to the protracted circumscriptions typical of the more philosophical or cultural discourses.But we have to read between the lines.It is only in passing that Gurevich presupposes an intention of an algorithm to compute a target.This seems in opposition to Deleuze and Guattari's (1983, p. 31) observation of the economy of machines, where "the product is always an offshoot of production, implanting itself upon it like a graft, and at the same time the parts of the machine are the fuel that makes it run", undoing any original intention and pointing exactly towards the de-emphasis of the subject/object distinction as indicated earlier.
The idea of excess and graft had also been employed by Jacques Derrida in his abstract notion of writing processes, and it reappears in Rheinberger's (1994) experimental systems.These are systems for the production of novelty, governed by a specific experimental culture and by their own operator-time.What is at play here is a dialectic of technical objects -previously stabilised sub-routines that could perhaps be identified with algorithms as intended and target-producing formalisms -and epistemic thingsthe articulation of traces (Rheinberger, 1998, p. 295) that "represent" that which does not yet have a reference.The anatomy of experimental systems may be useful for the understanding of what experimenting with algorithms implies, and if we carry this dialectic over to algorithmic art, the aesthetic object perhaps appears as the analogy of the epistemic object and arises through the articulation of traces or through graphematic excess in the course of experimentation and artistic practice.
It is also useful to think of this dialectic in the more general terms of medium/form as used by Luhmann.Then the algorithms as technical objects constitute the forms that articulate an otherwise intangible space or medium of algorithmicity, which is not the space of algorithm theory but the space of algorithmic agency (the principles that animate algorithms).Our endeavour as artists then is not to excel at the stabilisation and purification of algorithms as routines, but to explore and mark the space of algorithmicity, requiring indeed an effort to prevent stabilisation: "As soon as [an experimental system] settles on self-oscillation, its capability is reduced to a mere demonstration of itself-within a testand it has lost its research function.In order to pre-empt such oscillation and therefore release from the research front, the technical parts of an experimental system are permanently worked on and tinkered with."(Rheinberger, 2011, p. 69) [1] It resembles Deleuze and Guattari's statement that "the parts of the machine are the fuel that makes it run", however Rheinberger makes it clear that the most important aspect here is indeed the interface between the researcher and their apparatus.The critical agency that produces the contours of the epistemic thing is situated at this interface, as another instance of grafting or boundary crossing to the inside, i.e. the incursion of the machine into the researcher and vice versa.This partial revocation of boundaries between human and machine is what Rheinberger (2013, p. 199) calls extimacy, and what Barad calls intra-action (as cited in Kleinman, 2012), the idea that knowing requires proximity and entanglement.

| METHOD
To study algorithmic agency, being artists-researchers involved in the very practices at stake is not a hindrance, but almost a prerequisite.But we must find a strategy of demonstration, by which we can verbalise and communicate something about our entangled experiences.The material traces come to our assistance as the perhaps less subjective accounts of these intra-actions.The chosen method is an examination of how forms move between a number of specific sound works.Agency is then indicated by the, perhaps small, differences from iteration to iteration, and by getting implicated in material explorations we approach the unstable concept.
In the study of algorithmicity, the equivalent of the experimental system's laboratory ensemble I want to define as the configuration of an algorithm.The term configuration denotes a set of elements and their relations, stressing the heterogeneity of the elements involved which extend beyond the narrow set of procedure, data structure, control structure, etc. to include the peculiar ways they are framed and interact with more remotely positioned "environmental" elements.The action of tinkering is then subsumed under acts of reconfiguration, a more symmetric term that includes the possibility of initiation or catalysis through the speculative quality of algorithms.By carrying out this demonstration, it is proposed that artistic strategies can be built based on such motions that cut across the alleged boundaries of "pieces", and that this mode of communication may enrich the methodological repertoire of artistic research.It will lead to future questions guiding the study of algorithmic agency, such as:

•
What does experimentation mean in the context of algorithms?How does one arrive at algorithms and algorithmic descriptions, how do they obtain their form, how do they change form?How can experimental systems be used to build bridges between material practices and written or verbal descriptions?

•
What are the boundaries between human and machine, what are the consequences for authorship and intention?What is the structure of decisionmaking processes?What are the extent, kind, origin and goals of mechanisms of control?

•
What is the relationship between algorithms and bodies?Are they in opposition, or is there even something that could be called an algorithmic body or algorithmic corporeality?If so, how is it constituted, and how is our notion of corporeality being altered by programmabilities?

•
Which critical capacities are afforded by algorithms?What ways of re-appropriation do they offer, how are they epistemically and aesthetically charged?

•
What are the temporal, spatial and performative properties afforded by algorithms?

•
Are there strategic overlaps between an algorithmically-informed, reflexive sound and media art practice, and artistic and scientific research?

•
How does one address the tension between particularity and generalisability?How does one condition and preserve traces of algorithmic developments for future artistic and scientific practices?Which questions of notation, translation, representation, re-actualisation arise?

| MARKING A TRAJECTORY
The pieces for this case study have been chosen based on their use of similarities or imperfect reconstructions [2].Imperfect reconstruction I understand as a strategy that makes a continuous effort to rebuild a particular sound, structure or situation, where the aesthetic element lies in the foregrounding of process-immanent differences from iteration to iteration, and where no specific terminal state is preselected.It is the nature of any such trajectory that one can only artificially determine its starting and end point, since subjecting the chosen pieces or aspects thereof to the selected criteria is a constructive act.
The use of sound similarity approaches to elucidate the dance of agency between composer and computer had previously been investigated (Rutz, 2012), and the three pieces included in that study will be taken as points of departure from which we can now iterate with a more precise toolbox.The idea to work with similarity can be traced back to the notion of "sound mobile" (cf.Rutz, 2014b, p. 112), i.e. a structure that both guarantees an identity and objectform (recognition) but also produces ever varying changes so that the object is always experienced from different angles.That is to say, the notation is fixed but the performance is variable.
For example, we may have a description: "In the second section, lasting between one and two minutes, the recording of the sound of a rock sliding repeatedly across the floor of the room is heard."Then what I am interested in is that as each visitor or audience member is exposed to the piece, the same but different sliding rock is produced, for instance by selecting a slice from a much longer recording of these sounds.This is illustrated in Figure 2. The concept does not stand on its own, the listening process is irreducible.This is an extremely simple algorithm, making it a good case to start with.If we return to Kowalski's definition, the agency of algorithms is closely related to the exchange processes with their "environment", what has been labelled data, parameters and implementations in Figure 1.For example, the duration parameter will be the result of experimenting with an implementation of the algorithm, and on a finer grid we will encounter more parameters.Then the distribution function (linear, exponential, …) to choose randomly from the duration interval may be written into the control component or may be visible as a parameter from the outside.
Iterations may happen at the local level, within a piece.Observing them requires either strong discipline while composing or a second instance, conflicting with the "extimate" unity of artist and computer.A solution is to make this second instance an automatic tracing system integrated into the apparatus, an attempt that I have undertaken with a software framework (Rutz, 2014a).But iterations also happen when we move from piece to piece, as the boundaries of pieces are organisational demarcations providing useful gaps that may bring the re-entry of algorithms to the front, to use Spencer-Brown's term for the production of forms (1969/1979).Through these gaps we may then detect the medium in between the pieces, as drafted in Figure 3.So if we cross such a gap, the "sound mobile" is instantiated again in the live-electronic piece Inter-Play / Re-Sound (2011).One reconfiguration here is that instead of an existing corpus, live material is captured from a microphone.The function that preserves the piece's identity is fulfilled by analysing this material in terms of its spectral content, aligning it with precomposed structures for specific types of detected sounds or condensing the live signal into various buffers by keeping only those chunks that are similar to a template sound.The stable handle of "similarity" as our compositional strategy now introduces new algorithmic elements that may connect to existing ones.Here, such an element is the signal process that extracts and cross-correlates the spectral content.In other words, an important aspect of algorithmicity is the ability to compose algorithms [3].
Across the next gap, the automatism that was introduced through the spectral analysis provides the basis for the fixed media piece Leere Null (2012).Here we leave the metaphor of the sound mobile, and similarity -now taken as a centrifugal force away from identity -is used as a motor to produce unforeseen  sequences of sounds, scanning a huge corpus of heterogeneous input sounds.Figure 4 shows a timeline view of the second part of the piece.Even without seeing the detail of the spectra, one can grasp the ability of a simple proposition to organise the material.Here the algorithm is duplicated and follows two strategies for the selection of sounds: while subsequent sounds are always chosen based on strong similarity to one another (now taking into account both spectral and temporal development), one strategy tries to equally maximise similarity between concurrently heard sounds and the other tries to maximise dissimilarity.The result is two different forms, one defined by a coherence with concurrent sounds contributing to a fused spatial gestalt, the other defined by a Tudoresque ecology in which we can perceive many different elements transparently interpenetrating each other.
One simple switch in polarity leads to two irrelative qualities.What is the source of this irrelativity?It must lie outside the binary polarity switch.A plausible explanation is that we witness a phenomenon according to Barad's definition, that is the emergence of an entity through the interweaving of observations and the whole experimental arrangement.It would be false to simply attribute it to the complex interaction between individual components of the Kowalski algorithm, such as the specific type of spectral feature vector or the weighting function between subsequent and concurrent similarity.Instead it must be understood as the result of our experimentation with the algorithms, the whole trajectory, our investment that led to the particular constellation that we take now as its end point.Trajectory means we can always take another step: How the corpus of possible sounds came into existence.How sounds are "drained" from the corpus once they have been used, injecting thus a tiny reactive component into the algorithm [4].
The next node in the constructed trajectory is a joint.The first source is the previously used similarity measurement function based on audio feature extraction.The second source stems from my engagement with genetic algorithms (GA) in a research project on instrumental algorithmic composition.The potential of GA as asymptotic form generators is fascinating, but I was looking for a way to employ them in the domain of electronic music and sound art.I was interested in approaching synthetic sounds as a new material I had rejected so far.I began experimenting with the genetic programming (GP) of synthetic sounds by evolving a graph of signal processing blocks (UGens) and evaluating it based on the audio similarity with a given target sound.The convergence of this process is incredibly slow if the number of UGens is large and no combinatorial heuristics are given, as the space of possible graph topologies is immense.Instead of introducing such heuristics and constraints to shrink the solution space, I decided to observe the intermediate products of the search irrespective of their proximity to the target sound.
Several interesting aesthetic properties appear: There is a multitude and variety within the individuals of each population, especially considering those "less fit".The algorithm exhibits its peculiarities through the way DSP blocks are mutated and combined.Although the target sound has a fixed duration, the sound structures are temporally unlimited, moreover they are parametric models that can now be further composed and even extended to arbitrarily high numbers of channels.With this technique, I developed both a sound installation Configuration (2015) and a fixed medium piece Grenzwerte (2015).In the installation, the formal elements are triggered by and rendered through the specific space, the layout and atmosphere, the objects found in situ.Room recordings from the boat in which the installation took place were used to drive the genetic programming.But how to organise the sound individuals coming out of this programming?The algorithm is composed with two other algorithms.On the basis that an installation is foremost a spatial form with no musical-dramatic linearity, the individuals -all those falling within a given interval of fitness -were fed into a two-dimensional self-organising map (SOM) based on a spectral feature vector, providing a plane for look-up or traversal.But what kind of traversal?The classic Boids algorithm from Craig Reynolds provides a simple mechanism to scan a field with adjustable balance between coherence and disjointness.Each swarm particle picks up the sounds (using nearest neighbour search in the SOM) for one of the nine sounding objects in the installation.Often the particles are close to each other, producing perceptually close forms, sometimes the swarm breaks apart into groups, providing spatial contrast (Figure 5).In Grenzwerte, which is a stereo piece, the Boids did not make sense.The configuration was changed to unwind the map, beginning at one point and then finding the nearest neighbour, deleting the data point and repeating.One such path is shown in Figure 6.

| WEAVING TRAJECTORIES
Although the "similarity" trajectory could be developed with further nodes, the previous examples should suffice to understand the method.We can imagine the space of algorithmicity as an ether in which trajectories such as the one described before can be precipitated.A last important property to highlight is that these paths are not isolated fibres, but they are actually interwoven.Evidence will be given through a short second fibre that crosses the preceding one in the installation Configuration.This is a visual fibre, and therefore an illustration is helpful again to explicate what is difficult to verbalise (Figure 7).On the left side a rendering from a graphical user interface is shown that is used both to visualise processes during the development of a sound installation and to act as an interface for live improvisation.Both in the "unsupervised" display and the live operation, the problem of screen space occurs.The number of processes and their parameter structure change, so an intelligent partition of the display is sought.We also need to be able to pan and zoom at different levels of detail.The system derives from an information visualisation toolkit and a particular force-directed layout algorithm for graphs.An N-body force defines the gravity or repulsion between all vertices along with a Barnes-Hut coefficient, a drag force simulates friction, and a spring force controls the edges in the graph.As a result, processes will self-organise their use of the screen estate.Together with a convex hull rendering for groups of parameters, the interface obtains an aesthetics resembling amoebae on a specimen slide [5].Ever since this system became operational, it was self-evident that the visual beauty constitutes an autonomous quality beyond the functional design.The gap we cross in the transition from the left to the middle image of Figure 7 is the withdrawal from the context of an auxiliary display and the construction of a pure video work.In Configuration, one of the rooms contains a video triptych.The image shown here is a frame from one of the three videos, following the evolution of selected sound structures as they mutate and crossbreed through the genetic programmingthe point of intersection of the two fibres.To procure a vertical alignment, experiments were conducted with the algorithm, finally arriving at two custom "torque" forces that bring the structure into the desired vertical layout.
The second and last gap shown here occurs within this installation piece.The third of the video triptych attempts to find a visual form for text.(The movement of text components through my work would indeed be yet another fibre that finds a crossing point here.)From the augmented layout used to produce the GP video, an experimental system was constructed where all parameters could be adjusted.Although just a dozen in number, the dynamics become very complex and the spectrum of possible shapes and gestures extremely large.The reconfiguration that took place encompasses the exchange of UGens for text letters as vertices, the addition of a second edge type for connecting text lines, and most importantly the introduction of key frame snapshots for the parameter set.Interpolations are performed between key frames while the interaction between the parameters is all but linear, bringing the structure from stable plateaus to clusters and chaotic oscillations and back again.

| CONCLUSION
I have suggested that the renewed interest in algorithms stems from a change in perspective away from an independently constituted domain governed purely by an inner logic and towards performative qualities that arise conjointly with their temporal and environmental embedding and the mutual writing processes between human and machine.Algorithms do not simply reflect the way we organise and formalise our cognition, nor are they autonomous technological determinations that configure the mental model of the "user" (cf.Manovich, 2013, p. 208).Similarly, algorithmic aesthetics are no longer defined primarily by an elegance in the programming or its perceptual correlate, but through the articulation of traces in the conjoint agency, as programmer/artist/ researcher and machine go together.The particular artistic objects produced relate to the abstract reality of algorithms insofar as they are at the same time irreducible and "representational" (products grafted onto the production).
If we move experimentation to the foreground, the boundary that extends beyond Kowalski's definition becomes the main focus.The differential motion of the boundaries of experimental systems is often described as a "tinkering" by the researcher.Acknowledging a stronger balance between human and machine, with the latter characterised by speculative reason, it is suggested that, more formally than tinkering, both sides engage in boundary operations that are best described as reconfiguration, operations where many elements and relations, representations and concepts remain intact but a few critically change.I propose that reconfiguration happens on several time scales and that it will be especially useful to extend the observation beyond individual pieces and instead look macroscopically at series of pieces.This way the gaps are amplified that allow studying the nature of marking the medium of algorithmicity, the space of algorithmic agency.Pieces become resting points; a stable identifier is selected, such as the "sound mobile", and the possible moves of algorithmicity are mapped out, such as the catenation or nesting of algorithms, the reimplementation of a concept, the movement from a formerly internal detail of a program to a more exposed governing position.This project is also an invitation to artist-researchers to participate in the mapping process, since experimental systems have likewise been identified as an insightful perspective on artistic research (Schwab, 2013).
[3] The creation of new machines through 'orientation' and 'composition' has been examined by Heinz von Foerster (1993/2003) and Dirk Baecker (1996).If we think of a duplication of Fig. 1, the two copies of the system will be rotated and asymmetrically connected through their respective environmental transitions.In our example, the data structure of the algorithm that makes a random time selection within a corpus is filled with data provided by the algorithm that analyses the spectral frames of the input sound.
[4] When Agostino Di Scipio analysed Xenakis' stochastic music, he noted an inability of selforganisation -without a reactive component, "the unexpected, the singularity of events, does not become a source of information and transformation" (Di Scipio, 1998).
[5] This impression is limited in the figure as it misses the animation.

Figure 2 |Figure 3 |
Figure 2 | A mobile sound structure (left) in the sound installation Zelle 148 (2006; right).From a virtual data pool of half an hour of recorded sound, the notated duration is between one and two minutes.One actual rendering of five minutes is shown in blue.Spikes indicate resting points.

Figure 4 |
Figure 4 | Timeline showing sonogram arrangement of sounds in Leere Null.Time passes horizontally and concurrent sounds are vertically distributed.

Figure 5 |
Figure 5 | Traversal of sounds (left) in Configuration (right).Each red dot is a sound synthesis graph obtained through GP and placed in the SOM.Superimposed is a swarm of nine agents scanning the map, each connected to a sound transducer in the installation.

Figure 6 |
Figure 6 | Section from a path unwinding in Grenzwerte.Each data point is visualised by the corresponding UGen graph, the colour gradient describes the temporal succession.

Figure 7 |
Figure7| Three nodes from a visual fibre.A control surface for sound processes and live improvisation interface using a force-directed layout (left), a visual translation of the genetic programming based on the same layout augmented with additional forces (middle), a decomposition of text using the augmented layout (right).