Visualizing digital media interactions : providing feedback on jam2jam AV performances

Instrumental music performance is a well-established case of real-time interaction with technology and, when extended to ensembles, of interaction with others. However, these interactions are fleeting and the opportunities to reflect on action is limited, even though audio and video recording has recently provided important opportunities in this regard. In this paper we report on research to further extend these reflective opportunities through the capture and visualization of gestural data collected during collaborative virtual performances; specifically using the digital media instrument Jam2jam AV and the specifically-developed visualization software Jam2jam AV Visualize. We discusses how such visualization may assist performance development and understanding. The discussion engages with issues of representation, authenticity of virtual experiences, intersubjectivity and wordless collaboration, and creativity support. Two usage scenarios are described showing that collaborative intent is evident in the data visualizations more clearly than in audio-visual recordings alone, indicating that the visualization of performance gestures can be an efficient way of identifying deliberate and co-operative performance behaviours.

An open-source approach to music education through jam2jam XO

This article examines the philosophy and practice of open-source technology in the development of the jam2jam XO software for the One Laptop Per Child (OLPC) computer. It explores how open-source software principles, pragmatist philosophy, improvisation and constructionist epistemologies are operationalized in the design and development of music software, and how such reflection reveals both the strengths and weaknesses of the open-source software development paradigm. An overview of the jam2jam XO platform, its development processes and music educational uses is provided and resulting reflections on the strengths and weaknesses of open-source development for music education are discussed. From an educational and software development perspective, the act of creating open-source software is shown to be a valuable enterprise, however, just because the source code, creative content and experience design are accessible and 'open' to be changed, does not guarantee that educational practices in the use of that software will change. Research around the development and use of jam2jam XO suggests that open-source software development principles can have an impact beyond software development and on to aspects of experience design and learning relationships.

Evolving interactions : agile design for networked media performance

Network Jamming systems provide real-time collaborative performance experiences for novice or inexperienced users. In this paper we will outline the interaction design considerations that have emerged during through evolutionary development cycles of the jam2jam Network Jamming software that employs generative techniques that require particular attention to the human computer relationship. In particular we describe the co-evolution of features and uses, explore the role of agile development methods in supporting this evolution, and show how the provision of a clear core capability can be matched with options for enhanced features support multi-levelled user experience and skill develop.

The educational affordances of generative media in arts education

Network Jamming systems provide real-time collaborative media performance experiences for novice or inexperienced users. In this paper we will outline the theoretical and developmental drivers for our Network Jamming software, called jam2jam. jam2jam employs generative algorithmic techniques with particular implications for accessibility and learning. We will describe how theories of engagement have directed the design and development of jam2jam and show how iterative testing cycles in numerous international sites have informed the evolution of the system and its educational potential. Generative media systems present an opportunity for users to leverage computational systems to make sense of complex media forms through interactive and collaborative experiences. Generative music and art are a relatively new phenomenon that use procedural invention as a creative technique to produce music and visual media. These kinds of systems present a range of affordances that can facilitate new kinds of relationships with music and media performance and production. Early systems have demonstrated the potential to provide access to collaborative ensemble experiences to users with little formal musical or artistic expertise.This presentation examines the educational affordances of these systems evidenced by field data drawn from the Network Jamming Project. These generative performance systems enable access to a unique kind of music/media’ ensemble performance with very little musical/ media knowledge or skill and they further offer the possibility of unique interactive relationships with artists and creative knowledge through collaborative performance. Through the process of observing, documenting and analysing young people interacting with the generative media software jam2jam a theory of meaningful engagement has emerged from the need to describe and codify how users experience creative engagement with music/media performance and the locations of meaning. In this research we observed that the musical metaphors and practices of ‘ensemble’ or collaborative performance and improvisation as a creative process for experienced musicians can be made available to novice users. The relational meanings of these musical practices afford access to high level personal, social and cultural experiences. Within the creative process of collaborative improvisation lie a series of modes of creative engagement that move from appreciation through exploration, selection, direction toward embodiment. The expressive sounds and visions made in real-time by improvisers collaborating are immediate and compelling. Generative media systems let novices access these experiences with simple interfaces that allow them to make highly professional and expressive sonic and visual content simply by using gestures and being attentive and perceptive to their collaborators. These kinds of experiences present the potential for highly complex expressive interactions with sound and media as a performance. Evidence that has emerged from this research suggest that collaborative performance with generative media is transformative and meaningful. In this presentation we draw out these ideas around an emerging theory of meaningful engagement that has evolved from the development of network jamming software. Primarily we focus on demonstrating how these experiences might lead to understandings that may be of educational and social benefit.

Network jamming : distributed performance using generative music

Generative music systems can be performed by manipulating the values of their algorithmic parameters, and their semi-autonomous nature provides an opportunity for coordinated interaction amongst a network of systems, a practice we call Network Jamming. This paper outlines the characteristics of this networked performance practice and discusses the types of mediated musical relationships and ensemble configurations that can arise. We have developed and tested the jam2jam network jamming software over recent years. We describe this system, draw from our experiences with it, and use it to illustrate some characteristics of Network Jamming.

Navigating technological contexts and experience design in music education

For some time we have jokingly referred to our network jamming research with jam2jam as ‘Switched on Orff’ (Brown, Sorensen and Dillon 2002; Dillon 2003; Dillon 2006; Dillon 2006; Brown and Dillon 2007). The connection with electronic music and Wendy Carlos’ classic work ‘Switched on Bach’ was obvious; we were using electronic music in schools and with children. The deeper connection with Orff however was about recognising that electronic music and instruments could have cultural values and knowledge embedded in their design and practice in same way as what has come to be known as the Orff method (Orff and Keetman 1958-66). However whilst the Orff method focuses upon Western art music perceptual framework electronic instruments have the potential to have more fluid musical environments and even to move to interdisciplinary study by including visual media. Whilst the Orff method focused on making sense of Western art music through experience electronic environments potentially can make sense of the world of multi media that pervades our lives.

Music is a Wordless Knowing of Others : Resilience in Virtual Ensembles

Music making affects relationships with self and others by generating a sense of belonging to a culture or ideology (Bamford, 2006; Barovick, 2001; Dillon & Stewart, 2006; Fiske, 2000; Hallam, 2001). Whilst studies from arts education research present compelling examples of these relationships, others argue that they do not present sufficiently validated evidence of a causal link between music making experiences and cognitive or social change (Winner & Cooper, 2000; Winner & Hetland, 2000a, 2000b, 2001). I have suggested elsewhere that this disconnection between compelling evidence and observations of the effects of music making are in part due to the lack of rigor in research and the incapacity of many methods to capture these experiences in meaningful ways (Dillon, 2006). Part of the answer to these questions about rigor and causality lay in the creative use of new media technologies that capture the results of relationships in music artefacts. Crucially, it is the effective management of these artefacts within computer systems that allows researchers and practitioners to collect, organize, analyse and then theorise such music making experiences.

Collaborative digital media performance with generative music systems

Music is inherently active and interactive. Like technologies before them, digital systems provide a range of enhanced music performance opportunities. In this paper we outline the educational advantages of ensemble performance in which generative media systems are integrated. As a concrete example, we focus on our work with the jam2jam system which uses generative music processes to enhance collaborative music making. We suggest that our research points toward a new class of activities that maintain the well established benefits of ensemble performance while adding cultural and pedagogical value by leveraging the capabilities and cachet of digital media practices.

The promise and possibilities of music, digital media and National Broadband Network

In this article I would like to examine the promise and possibilities of music, digital media and National Broadband Network. I will do this based on concepts that have emerged from a study undertaken by Professor Andrew Brown and I that categorise technologies into what we term representational technologies and technologies with agency

Experience design and interactive software in music education research

This paper examines the integration of computing technologies into music education research in a way informed by constructivism. In particular, this paper focuses on an approach established by Jeanne Bamberger, which the author also employs, that integrates software design, pedagogical exploration, and the building of music education theory. In this tradition, researchers design software and associated activities to facilitate the interactive manipulation of musical structures and ideas. In short, this approach focuses on designing experiences and tools that support musical thinking and doing. In comparing the work of Jean Bamberger with that of the author, this paper highlights and discusses issues of significance and identifies lessons for future research.

eBILITY : from tool use to partnerships

In this article we identify how computational automation achieved through programming has enabled a new class of music technologies with generative music capabilities. These generative systems can have a degree of music making autonomy that impacts on our relationships with them; we suggest that this coincides with a shift in the music-equipment relationship from tool use to a partnership. This partnership relationship can occur when we use technologies that display qualities of agency. It raises questions about the kinds of skills and knowledge that are necessary to interact musically in such a partnership. These are qualities of musicianship we call eBility. In this paper we seek to define what eBility might consist of and how consideration of it might effect music education practice. The 'e' in eBility refers not only to the electronic nature of computing systems but also to the ethical, enabling, experiential and educational dimensions of the creative relationship with technologies with agency. We hope to initiate a discussion around differentiating what we term representational technologies from those with agency and begin to uncover the implications of these ideas for music educators in schools and communities. We hope also to elucidate the emergent theory and practice that has enabled the development of strategies for optimising this kind of eBility where the tool becomes partner. The identification of musical technologies with agency adds to the authors’ list of metaphors for technology use in music education that previously included tool, medium and instrument. We illustrate these ideas with examples and with data from our work with the jam2jam interactive music system. In this discussion we will outline our experiences with jam2jam as an example of a technology with agency and describe the aspects of eBility that interaction with it promotes.

Meaningful musical engagement : the potential impact of generative music systems on learning experiences

In order to create music, the student must establish a relationship with the musical materials. In this thesis, I examine the capacity of a generative music system called jam2jam to offer individuals a virtual musical play-space to explore. I outline the development of an iteration of software development named jam2jam blue and the evolution of a games-like user interface in the research design that jointly revealed the nature of this musical exploration. The findings suggest that the jam2jam blue interface provided an expressive gestural instrument to jam and experience musicmaking. By using the computer as an instrument, participants in this study were given access to meaningful musical experiences in both solo and ensemble situations and the researcher is allowed a view of their development of a relationship with the musical materials from the perspective of the individual participants. Through an iterative software development methodology, pedagogy and experience design were created simultaneously. The research reveals the potential for the jam2jam software to be used as a reflective tool for feedback and assessment purposes. The power of access to ensemble music making is realised though the participants’ virtual experiences which are brought into their physical space by sharing their experience with others. It is suggested that this interaction creates an environment conducive to self-initiated learning in which music is the language of interaction. The research concludes that the development of a relationship between the explorer and the musical materials is subject to the collaborative nature of the interaction through which the music is experienced.