Improvisation in interactive music systems

This project investigates machine listening and improvisation in interactive music systems with the goal of improvising musically appropriate accompaniment to an audio stream in real-time. The input audio may be from a live musical ensemble, or playback of a recording for use by a DJ. I present a collection of robust techniques for machine listening in the context of Western popular dance music genres, and strategies of improvisation to allow for intuitive and musically salient interaction in live performance. The findings are embodied in a computational agent – the Jambot – capable of real-time musical improvisation in an ensemble setting. Conceptually the agent’s functionality is split into three domains: reception, analysis and generation. The project has resulted in novel techniques for addressing a range of issues in each of these domains. In the reception domain I present a novel suite of onset detection algorithms for real-time detection and classification of percussive onsets. This suite achieves reasonable discrimination between the kick, snare and hi-hat attacks of a standard drum-kit, with sufficiently low-latency to allow perceptually simultaneous triggering of accompaniment notes. The onset detection algorithms are designed to operate in the context of complex polyphonic audio. In the analysis domain I present novel beat-tracking and metre-induction algorithms that operate in real-time and are responsive to change in a live setting. I also present a novel analytic model of rhythm, based on musically salient features. This model informs the generation process, affording intuitive parametric control and allowing for the creation of a broad range of interesting rhythms. In the generation domain I present a novel improvisatory architecture drawing on theories of music perception, which provides a mechanism for the real-time generation of complementary accompaniment in an ensemble setting. All of these innovations have been combined into a computational agent – the Jambot, which is capable of producing improvised percussive musical accompaniment to an audio stream in real-time. I situate the architectural philosophy of the Jambot within contemporary debate regarding the nature of cognition and artificial intelligence, and argue for an approach to algorithmic improvisation that privileges the minimisation of cognitive dissonance in human-computer interaction. This thesis contains extensive written discussions of the Jambot and its component algorithms, along with some comparative analyses of aspects of its operation and aesthetic evaluations of its output. The accompanying CD contains the Jambot software, along with video documentation of experiments and performances conducted during the project.

Discussions in space : interactive urban screens for enhancing citizen engagement

Local governments struggle to engage time poor and seemingly apathetic citizens, as well as the city's young digital natives, the digital locals. Capturing the attention of this digitally literate community who are technology and socially savvy adds a new quality to the challenge of community engagement for urban planning. This project developed and tested a lightweight design intervention towards removing the hierarchy between those who plan the city and those who use it. The aim is to narrow this gap by enhancing people's experience of physical spaces with digital, civic technologies that are directly accessible within that space. The study's research informed the development of a public screen system called Discussions In Space (DIS). It facilitates a feedback platform about specific topics, e.g., a concrete urban planning project, and encourages direct, in-situ, real-time user responses via SMS and Twitter. The thesis presents the findings of deploying and integrating DIS in a wide range of public and urban environments, including the iconic urban screen at Federation Square in Melbourne, to explore the Human-Computer Interaction (HCI) related challenges and implications. It was also deployed in conjunction with a major urban planning project in Brisbane to explore the system's opportunities and challenges of better engaging with Australia's new digital locals. Finally, the merits of the short-texted and ephemeral data generated by the system were evaluated in three focus groups with professional urban planners. DIS offers additional benefits for civic participation as it gives voice to residents who otherwise would not be easily heard. It also promotes a positive attitude towards local governments and gathers complementary information that is different than that captured by more traditional public engagement tools.

The social car : new interactive vehicular applications derived from social media and urban informatics

Digital information that is place- and time-specific, is increasingly becoming available on all aspects of the urban landscape. People (cf. the Social Web), places (cf. the Geo Web), and physical objects (cf. ubiquitous computing, the Internet of Things) are increasingly infused with sensors, actuators, and tagged with a wealth of digital information. Urban informatics research explores these emerging digital layers of the city at the intersection of people, place and technology. However, little is known about the challenges and new opportunities that these digital layers may offer to road users driving through today’s mega cities. We argue that this aspect is worth exploring in particular with regards to Auto-UI’s overarching goal of making cars both safer and more enjoyable. This paper presents the findings of a pilot study, which included 14 urban informatics research experts participating in a guided ideation (idea creation) workshop within a simulated environment. They were immersed into different driving scenarios to imagine novel urban informatics type of applications specific to the driving context.

Envisaging alternatives for practice : a study into the way concepts of sustain-ability can be explored within an interactive media arts practice

This practice-led study explores different ways the subject of sustain-ability can be addressed within an Interactive Media Arts practice. The exploration encompasses three creative projects, Charmed, Distracted and e. Menura superba. Grounded in an ecological philosophy inspired by vegetarianism and the critical design philosophy of defuturing, the work shows how such a philosophical position can guide the redirection of practice. The concern for sustain-ability within my practice, and more generally the question of Interactive Media Arts and sustain-ability, I refer to as a problématique. The objective of this study is not one of finding an answer or a truth to an instrumentally posed question, but to explore the complexities of the problématique through a program of practice and intellectual investigation. The aim being to redirect my practice and to find a renewed raison d’être for practice through a process of opening up, encountering, and discovering otherwise unknown possibilities for practice. In the context of sustain-ability, this opening up of possibilities can be considered a form of futuring. A futuring I argue is only possible if the things we take for granted as integral aspects of our being, practices and life worlds, are revealed in ways that estrange them, rendering them visible in ways that allow questioning and change.

Interactive product browsing and configuration using remote augmented reality sales services

Real-time remote sales assistance is an underdeveloped component of online sales services. Solutions involving web page text chat, telephony and video support prove problematic when seeking to remotely guide customers in their sales processes, especially with configurations of physically complex artefacts. Recently, there has been great interest in the application of virtual worlds and augmented reality to create synthetic environments for remote sales of physical artefacts. However, there is a lack of analysis and development of appropriate software services to support these processes. We extend our previous work with the detailed design of configuration context services to support the management of an interactive sales session using augmented reality. We detail the context and configuration services required, presenting a novel data service streaming configuration information to the vendor for business analytics. We expect that a fully implemented configuration management service, based on our design, will improve the remote sales experience for both customers and vendors alike via analysis of the streamed information.

Positive emotional experiences with portable interactive products

Positive emotions are central to human life and have implications to the overall quality of people's life (Fredrickson, 1998). This paper reports on positive experiences with two types of portable interactive devices (PIDs), specifically media/entertainment and medical/health devices. The study is based on a six-month longitudinal study exploring people's emotional experience and how PIDs mediate these experiences in everyday contexts. Previous findings by the authors (Gomez, Popovic & Blackler, 2011) presented four categories of activities including Feature, Functional, Mediation and Auxiliary activities and their relationship to emotional experience. The paper presents emotional experiences with specific activities reported with a focus on positive emotions. It concludes with a discussion of the findings on positive experiences and the implications for the future design of PIDs.

Analyzing the factors influencing the successful design and uptake of interactive systems to support social networks in urban neighborhoods

In urban residential environments in Australia and other developed countries, Internet access is on the verge of becoming a ubiquitous utility like gas or electricity. From an urban sociology and community informatics perspective, this article discusses new emerging social formations of urban residents that are based on networked individualism and the potential of Internet-based systems to support them. It proposes that one of the main reasons for the disappearance or nonexistence of urban residential communities is a lack of appropriate opportunities and instruments to encourage and support local interaction in urban neighborhoods. The article challenges the view that a mere reappropriation of applications used to support dispersed virtual communities is adequate to meet the place and proximity-based design requirements that community networks in urban neighborhoods pose. It argues that the key factors influencing the successful design and uptake of interactive systems to support social networks in urban neighborhoods include the swarming social behavior of urban dwellers; the dynamics of their existing communicative ecology; and the serendipitous, voluntary, and place-based quality of interaction between residents on the basis of choice, like-mindedness, mutual interest and support needs. Drawing on an analysis of these factors, the conceptual design framework of a prototype system — the urban tribe incubator — is presented.

Resilience by design: A participatory approach to designing an interactive digital application for promoting children's resilience

The promotion of resilience (the capacity of an individual or community to bounce back and recover from adversity) has become an important area of public health. In recent years it has expanded into the digital domain, and many online applications have been developed to promote children's resilience. In this study, it is argued that the majority of existing applications are limited because they take a didactic approach, and conceive of interaction as providing navigational choices. Because they simply provide information about resilience or replicate offline, scenario-based strategies, the understanding of resilience they provide is confined to a few, predetermined factors. In this study I propose a new, experiential approach to promoting resilience digitally. I define resilience as an emergent, situated and context-specific phenomenon. Using a Participatory Design model in combination with a salutogenic (strength-based) health methodology, this project has involved approximately 50 children as co-designers and co-researchers over two years. The children have contributed to the design of a new set of interactive resilience tools, which facilitate resilience promotion through dialogic and experiential learning. The major outcomes of this study include a new methodology for developing digital resilience tools, a new set of tools that have been developed and evaluated in collaboration with children and a set of design principles to guide future development. Beyond these initial and tangible outcomes, this study has also established that the benefits of introducing Participatory Design into a health promoting model rests primarily in the change of the role of children from "users" of technology and education to co-designers, where they assume a leadership role in both designing the tools and in directing their resilience learning.

Randomised controlled trial of an automated, interactive telephone intervention (TLC Diabetes) to improve type 2 diabetes management : Baseline findings and six-month outcomes

Background: Effective self-management of diabetes is essential for the reduction of diabetes-related complications, as global rates of diabetes escalate. Methods: Randomised controlled trial. Adults with type 2 diabetes (n = 120), with HbA1c greater than or equal to 7.5 %, were randomly allocated (4 × 4 block randomised block design) to receive an automated, interactive telephone-delivered management intervention or usual routine care. Baseline sociodemographic, behavioural and medical history data were collected by self-administered questionnaires and biological data were obtained during hospital appointments. Health-related quality of life (HRQL) was measured using the SF-36. Results: The mean age of participants was 57.4 (SD 8.3), 63 % of whom were male. There were no differences in demographic, socioeconomic and behavioural variables between the study arms at baseline. Over the six-month period from baseline, participants receiving the Australian TLC (Telephone-Linked Care) Diabetes program showed a 0.8 % decrease in geometric mean HbA1c from 8.7 % to 7.9 %, compared with a 0.2 % HbA1c reduction (8.9 % to 8.7 %) in the usual care arm (p = 0.002). There was also a significant improvement in mental HRQL, with a mean increase of 1.9 in the intervention arm, while the usual care arm decreased by 0.8 (p = 0.007). No significant improvements in physical HRQL were observed. Conclusions: These analyses indicate the efficacy of the Australian TLC Diabetes program with clinically significant post-intervention improvements in both glycaemic control and mental HRQL. These observed improvements, if supported and maintained by an ongoing program such as this, could significantly reduce diabetes-related complications in the longer term. Given the accessibility and feasibility of this kind of program, it has strong potential for providing effective, ongoing support to many individuals with diabetes in the future.

Exploration and reflection in interactive art : glass pond

Glass Pond is an interactive artwork designed to engender exploration and reflection through an intuitive, tangible interface and a simulation agent. It is being developed using iterative methods. A study has been conducted with the aim of illuminating user experience, interface, design, and performance issues.The paper describes the study methodology and process of data analysis including coding schemes for cognitive states and movements. Analysis reveals that exploration and reflection occurred as well as composing behaviours (unexpected). Results also show that participants interacted to varying degrees. Design discussion includes the artwork's (novel) interface and configuration.

Emergence in interactive art

This thesis is concerned with creating and evaluating interactive art systems that facilitate emergent participant experiences. For the purposes of this research, interactive art is the computer based arts involving physical participation from the audience, while emergence is when a new form or concept appears that was not directly implied by the context from which it arose. This emergent ‘whole’ is more than a simple sum of its parts. The research aims to develop understanding of the nature of emergent experiences that might arise during participant interaction with interactive art systems. It also aims to understand the design issues surrounding the creation of these systems. The approach used is Practice-based, integrating practice, evaluation and theoretical research. Practice used methods from Reflection-in-action and Iterative design to create two interactive art systems: Glass Pond and +-now. Creation of +-now resulted in a novel method for instantiating emergent shapes. Both art works were also evaluated in exploratory studies. In addition, a main study with 30 participants was conducted on participant interaction with +-now. These sessions were video recorded and participants were interviewed about their experience. Recordings were transcribed and analysed using Grounded theory methods. Emergent participant experiences were identified and classified using a taxonomy of emergence in interactive art. This taxonomy draws on theoretical research. The outcomes of this Practice-based research are summarised as follows. Two interactive art systems, where the second work clearly facilitates emergent interaction, were created. Their creation involved the development of a novel method for instantiating emergent shapes and it informed aesthetic and design issues surrounding interactive art systems for emergence. A taxonomy of emergence in interactive art was also created. Other outcomes are the evaluation findings about participant experiences, including different types of emergence experienced and the coding schemes produced during data analysis.

An interactive and adaptive building layer : strategies for allowing people to become advanced building-users

This paper reports outcomes of a pilot study to develop a conceptual framework to allow people to retrofit a building-layer to gain better control of their own built- environments. The study was initiated by the realisation that discussions surrounding the improvement of building performances tend to be about top-down technological solutions rather than to help and encourage bottom-up involvement of building-users. While users are the ultimate beneficiaries and their feedback is always appreciated, their direct involvements in managing buildings would often be regarded as obstruction or distraction. This is largely because casual interventions by uninformed building-users tend to disrupt the system. Some earlier researches showed however that direct and active participation of users could improve the building performance if appropriate training and/or systems were introduced. We also speculate this in long run would also make the built environment more sustainable. With this in mind, we looked for opportunities to retrofit our own office with an interactive layer to study how we could introduce ad-hoc systems for building-users. The aim of this paper is to describe our vision and initial attempts followed by discussion.

Interactive conference voting

We describe and analyze opinion polling results from interactive voting procedures undertaken before and after presentations during the Outcome Measures in Rheumatoid Arthritis Clinical Trials Conference (OMERACT II) in Ottawa, Canada, June 30-July 2, 1994. The scoring procedure was a matched voting design; when a participant used the same keypad at the beginning and end of voting, change within a participant could be estimated. Participants, experienced in the rheumatic diseases included clinicians, researchers, methodologists, regulators, and representatives of the pharmaceutical industry. Patients under consideration were those with any rheumatic diseases. Questions were constructed to evaluate the change in voting behavior expected from the content of the presentation. Statistically significant and substantively important changes were evident in most questions.

Learning hierarchical prototypes of motion time series for interactive systems

Abstract. For interactive systems, recognition, reproduction, and generalization of observed motion data are crucial for successful interaction. In this paper, we present a novel method for analysis of motion data that we refer to as K-OMM-trees. K-OMM-trees combine Ordered Means Models (OMMs) a model-based machine learning approach for time series with an hierarchical analysis technique for very large data sets, the K-tree algorithm. The proposed K-OMM-trees enable unsupervised prototype extraction of motion time series data with hierarchical data representation. After introducing the algorithmic details, we apply the proposed method to a gesture data set that includes substantial inter-class variations. Results from our studies show that K-OMM-trees are able to substantially increase the recognition performance and to learn an inherent data hierarchy with meaningful gesture abstractions.