Towards an intelligent score following system: Handling of mistakes and jumps encountered during piano practicing

Score following has been an important area of research in AI and music since the mid 80's. Various systems were developed, but they were predominantly for providing automated accompaniment to live concert performances, dealing mostly with issues relating to pitch detection and identification of embellished melodies. They have a big potential in the area of education where student performers benefit in practice situations. Current accompaniment systems are not designed to deal with errors that may occur during practising. In this paper we present a system developed to provide accompaniment for students practising at home. First a survey of score following will be given. Then the capabilities of the system will be explained, and the results from the first experiments of the monophonic score following system will be presented.

Progressive Audio Scrambling in Compressed Domain

Audio scrambling can be employed to ensure confidentiality in audio distribution. We first describe scrambling for raw audio using the discrete wavelet transform (DWT) first and then focus on MP3 audio scrambling. We perform scrambling based on a set of keys which allows for a set of audio outputs having different qualities. During descrambling, the number of keys provided and the number of rounds of descrambling performed will decide the audio output quality. We also perform scrambling by using multiple keys on the MP3 audio format. With a subset of keys, we can descramble to obtain a low quality audio. However, we can obtain the original quality audio by using all of the keys. Our experiments show that the proposed algorithms are effective, fast, simple to implement while providing flexible control over the progressive quality of the audio output. The security level provided by the scheme is sufficient for protecting MP3 music content.

Multimodal Biometric Human Recognition for Perceptual Human-Computer Interaction

In this paper, a novel video-based multimodal biometric verification scheme using the subspace-based low-level feature fusion of face and speech is developed for specific speaker recognition for perceptual human--computer interaction (HCI). In the proposed scheme, human face is tracked and face pose is estimated to weight the detected facelike regions in successive frames, where ill-posed faces and false-positive detections are assigned with lower credit to enhance the accuracy. In the audio modality, mel-frequency cepstral coefficients are extracted for voice-based biometric verification. In the fusion step, features from both modalities are projected into nonlinear Laplacian Eigenmap subspace for multimodal speaker recognition and combined at low level. The proposed approach is tested on the video database of ten human subjects, and the results show that the proposed scheme can attain better accuracy in comparison with the conventional multimodal fusion using latent semantic analysis as well as the single-modality verifications. The experiment on MATLAB shows the potential of the proposed scheme to attain the real-time performance for perceptual HCI applications.

The Pontydian Performance:the performative layer

In this paper we reflect on the performer-instrument relationship by turning towards the thinking practices of the French philosopher Maurice Merleau-Ponty (1908-1961). Merleau-Ponty's phenomenological idea of the body as being at the centre of the world highlights an embodied position in the world and bestows significance onto the body as a whole, onto the body as a lived body. In order to better understand this two-way relationship of instrument and performer, we introduce the notion of the performative layer, which emerges through strategies for dealing with discontinuities, breakdowns and the unexpected in network performance.

Impure Thinking Practices and Clinical Acts: The Sonorous Becomings of Heidi Fast

This article introduces the recent sound works of Heidi Fast, a Finnish voice and performance artist. Fast’s creative practice operates between art and philosophy, and articulates several ‘zones of becoming’: what Fast designates as ‘the clinical’, ‘the virtual’ and ‘vocal thought-material’. Using a methodology of routing, the article shows how these zones emerge as aesthetic, ethical and political concerns within Fast’s work. Since 2005, Fast’s sound works have variously taken shape as miniature concerts, social sculptures, imaginary soundscapes and environmental music performances. Drawing upon the writings of theorists who have helped shape her practice, this article argues that Fast uses sound and voice to propose an ‘actualising philosophy’. This philosophy actualises virtualities (unrealised potentials), affecting transformative shifts through tiny mutations in perceptions and behaviours.

Digital Musical Interactions: Performer-System Relationships and their Perception by Spectators

This article adopts an ecological view of digital musical interactions, first considering the relationship between performers and digital systems, and then spectators’ perception of these interactions. We provide evidence that the relationships between performers and digital music systems are not necessarily instrumental in the same was as they are with acoustic systems, and nor should they always strive to be. Furthermore, we report results of a study suggesting that spectators may not perceive such interactions in the same way as performances with musical instruments. We present implications for the design of digital musical interactions, suggesting that designers should embrace the reality that digital systems are malleable and dynamic, and may engage performers and spectators in different modalities, sometimes simultaneously.

Interview with Paul DeMarinis

Since the early 1970s, the American electronic media artist Paul DeMarinis (b. 1948, Cleveland, Ohio, USA) has created works that re-imagine modes of communication and reinvent the technologies that enable communication. His works (see Table 1) have taken shape as recordings, performances, electronic inventions, and site-specific and interactive installations; many are considered landmarks in the histories of electronic music and media art. Paul DeMarinis pioneered live performance with computers, collaborated on landmark works with artists like David Tudor and Robert Ashley, undertook several tours with the Merce Cunningham Dance Company, and brought to life obscure technologies such as the flame loudspeaker (featured in his 2004 sculpture Firebirds). His interactive installation The Music Room (1982), commissioned by Frank Oppenheimer for the Exploratorium in San Francisco, was the first automatic music work to reach a significant audience. His album Music As A Second Language (1991) marks one of the most extensive explorations of the synthesized voice and speech melodies to date. Installations like The Edison Effect (1989-1993), in which lasers scan ancient recordings to produce music, and The Messenger (1998/2005), in which electronic mail messages are displayed on alphabetic telegraph receivers, illustrate a creative process that Douglas Kahn (1994) has called "reinventing invention." [etc]

A Framework for the Evaluation of Digital Musical Instruments

At the outset of a discussion of evaluating digital musical instruments, that is to say instruments whose sound generators are digital and separable though not necessarily separate from their control interfaces (Malloch, 2006), it is reasonable to ask what the term evaluation in this context really means. After all, there may be many perspectives from which to view the effectiveness or otherwise of the instruments we build. For most performers, performance on an instrument becomes a means of evaluating how well it functions in the context of live music making, and their measure of success is the response of the audience to their performance. Audiences evaluate performances on the basis of how engaged they feel they have been by what they have seen and heard. When questioned, they are likely to describe good performances as “exciting,” “skillful,” “musical.” Bad performances are “boring,” and those which are marred by technical malfunction are often dismissed out of hand. If performance is considered to be a valid means of evaluating a musical instrument, then it follows that, for the field of DMI design, a much broader definition of the term “evaluation” than that typically used in human-computer interaction (HCI) is required to reflect the fact that there are a number of stakeholders involved in the design and evaluation of DMIs. In addition to players and audiences, there are also composers, instrument builders, component manufacturers, and perhaps even customers, each of whom will have a different concept of what is meant by “evaluation.”

Evocative Listening : Mediated Practices in Everyday Life

The history of sonic arts is charged with transgressive practices that seek to expose the social, aural and cultural thresholds across various listening experiences, posing new questions in terms of the dialogue between listener and place. Recent work in sonic art exposes the need for an experiential understanding of listening that foregrounds the use of new personal technologies, environmental philosophy and the subject–object relationship. This paper aims to create a vocabulary that better contextualises recent installations and performances produced within the context of everyday life, by researchers and artists at the Sonic Arts Research Centre at Queen's University Belfast.

Common-sense knowledge for a computer vision system for human action recognition

This work presents a novel approach for human action recognition based on the combination of computer vision techniques and common-sense knowledge and reasoning capabilities. The emphasis of this work is on how common sense has to be leveraged to a vision-based human action recognition so that nonsensical errors can be amended at the understanding stage. The proposed framework is to be deployed in a realistic environment in which humans behave rationally, that is, motivated by an aim or a reason. © 2012 Springer-Verlag.

Moving with Beats and Loops: The Structure of Auditory Events and Sensorimotor Timing

Traditionally, audio-motor timing processes have been understood as motor output from an internal clock, the speed of which is set by heard sound pulses. In contrast, this paper proposes a more ecologically-grounded approach, arguing that audio-motor processes are better characterized as performed actions on the perceived structure of auditory events. This position is explored in the context of auditory sensorimotor synchronization and continuation timing. Empirical research shows that the structure of sounds as auditory events can lead to marked differences in movement timing performance. The nature of these effects is discussed in the context of perceived action-relevance of auditory event structure. It is proposed that different forms of sound invite or support different patterns of sensorimotor timing. Hence, the temporal information in looped auditory signals is more than just the interval durations between onsets: all metronomes are not created equal. The potential implications for auditory guides in motor performance enhancement are also described.