Improvisation in interactive music systems

This project investigates machine listening and improvisation in interactive music systems with the goal of improvising musically appropriate accompaniment to an audio stream in real-time. The input audio may be from a live musical ensemble, or playback of a recording for use by a DJ. I present a collection of robust techniques for machine listening in the context of Western popular dance music genres, and strategies of improvisation to allow for intuitive and musically salient interaction in live performance. The findings are embodied in a computational agent – the Jambot – capable of real-time musical improvisation in an ensemble setting. Conceptually the agent’s functionality is split into three domains: reception, analysis and generation. The project has resulted in novel techniques for addressing a range of issues in each of these domains. In the reception domain I present a novel suite of onset detection algorithms for real-time detection and classification of percussive onsets. This suite achieves reasonable discrimination between the kick, snare and hi-hat attacks of a standard drum-kit, with sufficiently low-latency to allow perceptually simultaneous triggering of accompaniment notes. The onset detection algorithms are designed to operate in the context of complex polyphonic audio. In the analysis domain I present novel beat-tracking and metre-induction algorithms that operate in real-time and are responsive to change in a live setting. I also present a novel analytic model of rhythm, based on musically salient features. This model informs the generation process, affording intuitive parametric control and allowing for the creation of a broad range of interesting rhythms. In the generation domain I present a novel improvisatory architecture drawing on theories of music perception, which provides a mechanism for the real-time generation of complementary accompaniment in an ensemble setting. All of these innovations have been combined into a computational agent – the Jambot, which is capable of producing improvised percussive musical accompaniment to an audio stream in real-time. I situate the architectural philosophy of the Jambot within contemporary debate regarding the nature of cognition and artificial intelligence, and argue for an approach to algorithmic improvisation that privileges the minimisation of cognitive dissonance in human-computer interaction. This thesis contains extensive written discussions of the Jambot and its component algorithms, along with some comparative analyses of aspects of its operation and aesthetic evaluations of its output. The accompanying CD contains the Jambot software, along with video documentation of experiments and performances conducted during the project.

Improvisation, process drama and dramatic art

This paper re-conceptualises the notion of improvisation as it has developed in schools since the 1960s. It outlines the theoretical case for naming the distinctive improvisational practice that has emerged in schools - namely Process Drama. The paper sketches seven characteristics of Process Drama and then outlines the case for seeing it as an emerging but robust form of dramatic art. It concludes by arguing that drama teachers, in developing process drama, have created a new form of both education and art.

Promoting articulated action from diverse stakeholders in response to public policy scenarios: A case analysis of the use of ‘scenario improvisation’ method

In this paper we present a novel application of scenario methods to engage a diverse constituency of senior stakeholders, with limited time availability, in debate to inform planning and policy development. Our case study project explores post-carbon futures for the Latrobe Valley region of the Australian state of Victoria. Our approach involved initial deductive development of two ‘extreme scenarios’ by a multi-disciplinary research team, based upon an extensive research programme. Over four workshops with the stakeholder constituency, these initial scenarios were discussed, challenged, refined and expanded through an inductive process, whereby participants took ‘ownership’ of a final set of three scenarios. These were both comfortable and challenging to them. The outcomes of this process subsequently informed public policy development for the region. Whilst this process did not follow a single extant structured, multi-stage scenario approach, neither was it devoid of form. Here, we seek to theorise and codify elements of our process – which we term ‘scenario improvisation’ – such that others may adopt it.

A participatory design agenda for ubiquitous computing : a case study in dental practice

This paper reflects upon our attempts to bring a participatory design approach to design research into interfaces that better support dental practice. The project brought together design researchers, general and specialist dental practitioners, the CEO of a dental software company and, to a limited extent, dental patients. We explored the potential for deployment of speech and gesture technologies in the challenging and authentic context of dental practices. The paper describes the various motivations behind the project, the negotiation of access and the development of the participant relationships as seen from the researchers' perspectives. Conducting participatory design sessions with busy professionals demands preparation, improvisation, and clarity of purpose. The paper describes how we identified what went well and when to shift tactics. The contribution of the paper is in its description of what we learned in bringing participatory design principles to a project that spanned technical research interests, commercial objectives and placing demands upon the time of skilled professionals.

Education for all : providing curriculum for students with disabilities in a standards-based environment : implications for Queensland schools

This report investigates lessons learned by educators in the United States when providing a standards-based curriculum for all students including Students with Disabilities (SWD). Assumptions about implementation of these lessons are then made to the Queensland school system. Queensland mainstream schools currently provide a standards-based curriculum for over sixteen thousand-four hundred students with mild-moderate disabilities and appear to be challenged by this new educational reform and its implications to school and teacher practices, beliefs and attitudes. The analysis of US research, literature and educational policy for this report, has provided some implications for Queensland schools in the areas of student participation, achievement and curriculum planning to provide an “education for all”. The analysis and comparison of legislation and policy, which demonstrates some significant similarities, provides greater validity for the application of lessons learned in the United States to the Queensland context. The key findings about lessons learned provides Queensland schools with some assumptions as to why and how they need to refocus school leader and teachers’ practices, beliefs and attitudes to provide an “education for all”. These lessons infer that school leaders and teachers to explicitly focus on equity, expectation, accountability, performance, alignment and collaboration so that effective curriculum is provided for SWD, indeed all students, in the Queensland standards-based curriculum environment.

¡Latin jazz! : a syncretic journey from Spain, Cuba, the United States and back

The creative work, ¡Latin Jazz! is a 50 minute radio documentary to be broadcast on ABC Classic FM. It looks at the evolution of Latin jazz from Spain, Cuba and the United States. It examines the social effects on the style and specifically on the syncretic movement between the countries. The documentary traces my travel to Madrid, Spain and looks at Latin jazz through a deconstruction of the style, musical examples and interviews with prominent artists. Artists interviewed were Chano Domínguez, a Spanish flamenco jazz pianist, Bobby Martínez an American saxophonist, Alain Pérez a Cuban bassist and Pepe Rivero a Cuban pianist. The exegesis supports the radio documentary by examining the style in more depth, and is broken into three main sections. First it traces the historical relationship that occurred through the Ida y Vuelta (To and Fro), the similarities and influences through the habanera, the decíma and the religion of Santería. This is followed by specific musical elements within Latin jazz such as instrumentation, clave, harmony and improvisation, whilst the third section looks at the influences of the new syncretic movement back to Spain.

Do androids dream of electric chimera?

This paper describes the use of the Chimera Architecture as the basis for a generative rhythmic improvisation system that is intended for use in ensemble contexts. This interactive soft- ware system learns in real time based on an audio input from live performers. The paper describes the components of the Chimera Architecture including a novel analysis engine that uses prediction to robustly assess the rhythmic salience of the input stream. Analytical results are stored in a hierarchical structure that includes multiple scenarios which allow ab- stracted and alternate interpretations of the current metrical context. The system draws upon this Chimera Architecture when generating a musical response. The generated rhythms are intended to have a particular ambiguity in relation to the music performance by other members of the ensemble. Ambi- guity is controlled through alternate interpretations of the Chimera. We describe an implementation of the Chimera Ar- chitecture that focuses on rhythmic material, and present and discuss initial experimental results of the software system playing along with recordings of a live performance.

Eco-structure data forms : classification of data analysis using perceived design affordances for musical outcomes

This paper explores a method of comparative analysis and classification of data through perceived design affordances. Included is discussion about the musical potential of data forms that are derived through eco-structural analysis of musical features inherent in audio recordings of natural sounds. A system of classification of these forms is proposed based on their structural contours. The classifications include four primitive types; steady, iterative, unstable and impulse. The classification extends previous taxonomies used to describe the gestural morphology of sound. The methods presented are used to provide compositional support for eco-structuralism.

TraSe algorithm : automatic evolutionary morphing of electronic dance music

The TraSe (Transform-Select) algorithm has been developed to investigate the morphing of electronic music through automatically applying a series of deterministic compositional transformations to the source, guided towards a target by similarity metrics. This is in contrast to other morphing techniques such as interpolation or parameters or probabilistic variation. TraSe allows control over stylistic elements of the music through user-defined weighting of numerous compositional transformations. The formal evaluation of TraSe was mostly qualitative and occurred through nine participants completing an online questionnaire. The music generated by TraSe was generally felt to be less coherent than a human composed benchmark but in some cases judged as more creative.

A brush with the real world : the future of inertial motion capture in live performance

3D Motion capture is a medium that plots motion, typically human motion, converting it into a form that can be represented digitally. It is a fast evolving field and recent inertial technology may provide new artistic possibilities for its use in live performance. Although not often used in this context, motion capture has a combination of attributes that can provide unique forms of collaboration with performance arts. The inertial motion capture suit used for this study has orientation sensors placed at strategic points on the body to map body motion. Its portability, real-time performance, ease of use, and its immunity from line-of-sight problems inherent in optical systems suggest it would work well as a live performance technology. Many animation techniques can be used in real-time. This research examines a broad cross-section of these techniques using four practice-led cases to assess the suitability of inertial motion capture to live performance. Although each case explores different visual possibilities, all make use of the performativity of the medium, using either an improvisational format or interactivity among stage, audience and screen that would be difficult to emulate any other way. A real-time environment is not capable of reproducing the depth and sophistication of animation people have come to expect through media. These environments take many hours to render. In time the combination of what can be produced in real-time and the tools available in a 3D environment will no doubt create their own tree of aesthetic directions in live performance. The case study looks at the potential of interactivity that this technology offers.

The educational affordances of generative media in arts education

Network Jamming systems provide real-time collaborative media performance experiences for novice or inexperienced users. In this paper we will outline the theoretical and developmental drivers for our Network Jamming software, called jam2jam. jam2jam employs generative algorithmic techniques with particular implications for accessibility and learning. We will describe how theories of engagement have directed the design and development of jam2jam and show how iterative testing cycles in numerous international sites have informed the evolution of the system and its educational potential. Generative media systems present an opportunity for users to leverage computational systems to make sense of complex media forms through interactive and collaborative experiences. Generative music and art are a relatively new phenomenon that use procedural invention as a creative technique to produce music and visual media. These kinds of systems present a range of affordances that can facilitate new kinds of relationships with music and media performance and production. Early systems have demonstrated the potential to provide access to collaborative ensemble experiences to users with little formal musical or artistic expertise.This presentation examines the educational affordances of these systems evidenced by field data drawn from the Network Jamming Project. These generative performance systems enable access to a unique kind of music/media’ ensemble performance with very little musical/ media knowledge or skill and they further offer the possibility of unique interactive relationships with artists and creative knowledge through collaborative performance. Through the process of observing, documenting and analysing young people interacting with the generative media software jam2jam a theory of meaningful engagement has emerged from the need to describe and codify how users experience creative engagement with music/media performance and the locations of meaning. In this research we observed that the musical metaphors and practices of ‘ensemble’ or collaborative performance and improvisation as a creative process for experienced musicians can be made available to novice users. The relational meanings of these musical practices afford access to high level personal, social and cultural experiences. Within the creative process of collaborative improvisation lie a series of modes of creative engagement that move from appreciation through exploration, selection, direction toward embodiment. The expressive sounds and visions made in real-time by improvisers collaborating are immediate and compelling. Generative media systems let novices access these experiences with simple interfaces that allow them to make highly professional and expressive sonic and visual content simply by using gestures and being attentive and perceptive to their collaborators. These kinds of experiences present the potential for highly complex expressive interactions with sound and media as a performance. Evidence that has emerged from this research suggest that collaborative performance with generative media is transformative and meaningful. In this presentation we draw out these ideas around an emerging theory of meaningful engagement that has evolved from the development of network jamming software. Primarily we focus on demonstrating how these experiences might lead to understandings that may be of educational and social benefit.

The making and placing of The Escalators

The band The Escalators together with the music uniquely composed for it and a subsequent CD and DVDs was the work that emerged from my period of research. The areas of interest that were investigated were sampling, minimalism, stasis, the work of David Lynch, as well as a desire to produce new and innovative music. The above concepts defined the bands composition and makeup. While each may be regarded as a discreet concept with its own boundaries in my work they seamlessly intermingle resulting in that which is unique to The Escalators sound. The research methodology used for this work was practice led research.

Risky business : making decisions off the cuff

In line with accepted decision making theory, individuals engage in rational decision making. Decisions made under conditions of bounded rationality may have serious adverse consequences. Employees making decisions on behalf of their employer are often faced with situations where perfect and complete information is not available, and time is limited. Under such conditions, we posit that employees will make decisions that are increasingly bounded. At its most extreme neither time nor information is available to make a decision and rational decision making, bounded or not, reaches its limits. Many authors suggest that this is the point at which improvisation takes place. Although opinion in the literature is mixed regarding the efficacy of improvised decisions, we argue that improvised decisions place the organisation at considerable risk and as a consequence are undesirable.

That's a rap : the making of an Australian hip hop integrated musical film

To date, the majority of films that utilise or feature hip hop music and culture, have either been in the realms of documentary, or in ‘show musicals’ (where the film musical’s device of characters’ bursting into song, is justified by the narrative of a pursuit of a career in the entertainment industry). Thus, most films that feature hip hop expression have in some way been tied to the subject of hip hop. A research interest and enthusiasm was developed for utilising hip hop expression in film in a new way, which would extend the narrative possibilities of hip hop film to wider topics and themes. The creation of the thesis film Out of My Cloud, and the writing of this accompanying exegesis, investigates a research concern of the potential for the use of hip hop expression in an ‘integrated musical’ film (where characters’ break into song without conceit or explanation). Context and rationale for Out of My Cloud (an Australian hip hop ‘integrated musical’ film) is provided in this writing. It is argued that hip hop is particularly suitable for use in a modern narrative film, and particularly in an ‘integrated musical’ film, due to its: current vibrancy and popularity, rap (vocal element of hip hop) music’s focus on lyrical message and meaning, and rap’s use as an everyday, non-performative method of communication. It is also argued that Australian hip hop deserves greater representation in film and literature due to: its current popularity, and its nature as a unique and distinct form of hip hop. To date, representation of Australian hip hop in film and television has almost solely been restricted to the documentary form. Out of My Cloud borrows from elements of social realist cinema such as: contrasts with mainstream cinema, an exploration/recognition of the relationship between environment and development of character, use of non-actors, location-shooting, a political intent of the filmmaker, displaying sympathy for an underclass, representation of underrepresented character types and topics, and a loose narrative structure that does not offer solid resolution. A case is made that it may be appropriate to marry elements of social realist film with hip hop expression due to common characteristics, such as: representation of marginalised or underrepresented groups and issues in society, political objectives of the artist/s, and sympathy for an underclass. In developing and producing Out of My Cloud, a specific method of working with, and filming actor improvisation was developed. This method was informed by improvisation and associated camera techniques of filmmakers such as Charlie Chaplin, Mike Leigh, Khoa Do, Dogme 95 filmmakers, and Lars von Trier (post-Dogme 95). A review of techniques used by these filmmakers is provided in this writing, as well as the impact it has made on my approach. The method utilised in Out of My Cloud was most influenced by Khoa Do’s technique of guiding actors to improvise fairly loosely, but with a predetermined endpoint in mind. A variation of this technique was developed for use in Out of My Cloud, which involved filming with two cameras to allow edits from multiple angles. Specific processes for creating Out of My Cloud are described and explained in this writing. Particular attention is given to the approaches regarding the story elements and the music elements. Various significant aspects of the process are referred to including the filming and recording of live musical performances, the recording of ‘freestyle’ performances (lyrics composed and performed spontaneously) and the creation of a scored musical scene involving a vocal performance without regular timing or rhythm. The documentation of processes in this writing serve to make the successful elements of this film transferable and replicable to other practitioners in the field, whilst flagging missteps to allow fellow practitioners to avoid similar missteps in future projects. While Out of My Cloud is not without its shortcomings as a short film work (for example in the areas of story and camerawork) it provides a significant contribution to the field as a working example of how hip hop may be utilised in an ‘integrated musical’ film, as well as being a rare example of a narrative film that features Australian hip hop. This film and the accompanying exegesis provide insights that contribute to an understanding of techniques, theories and knowledge in the field of filmmaking practice.

48 Hour Game Making Challenge - In Progress [Four Episodes]

Statement: Jams, Jelly Beans and the Fruits of Passion Let us search, instead, for an epistemology of practice implicit in the artistic, intuitive processes which some practitioners do bring to situations of uncertainty, instability, uniqueness, and value conflict. (Schön 1983, p40) Game On was born out of the idea of creative community; finding, networking, supporting and inspiring the people behind the face of an industry, those in the mist of the machine and those intending to join. We understood this moment to be a pivotal opportunity to nurture a new emerging form of game making, in an era of change, where the old industry models were proving to be unsustainable. As soon as we started putting people into a room under pressure, to make something in 48hrs, a whole pile of evolutionary creative responses emerged. People refashioned their craft in a moment of intense creativity that demanded different ways of working, an adaptive approach to the craft of making games – small – fast – indie. An event like the 48hrs forces participants’ attention on the process as much as the outcome. As one game industry professional taking part in a challenge for the first time observed: there are three paths in the genesis from idea to finished work: the path that focuses on mechanics; the path that focuses on team structure and roles and the path that focuses on the idea, the spirit – and the more successful teams need to put the spirit of the work first and foremost. The spirit drives the adaptation, it becomes improvisation. As Schön says: “Improvisation consists on varying, combining and recombining a set of figures within the schema which bounds and gives coherence to the performance.” (1983, p55). This improvisational approach is all about those making the games: the people and the principles of their creative process. This documentation evidences the intensity of their passion, determination and the shit that they are prepared to put themselves through to achieve their goal – to win a cup full of jellybeans and make a working game in 48hrs. 48hr is a project where, on all levels, analogue meets digital. This concept was further explored through the documentation process. This set of four videos were created by Cameron Owen on the fly during the challenge using both the iphone video camera and editing software in order to be available with immediacy and allow the event audience to share the experience - and perhaps to give some insights into the creative process exposed by the 48 hour challenge. ____________________________ Schön, D. A. 1983, The Reflective Practitioner: How Professionals Think in Action, Basic Books, New York