Beat that word : How listeners integrate beat gesture and focus in multimodal speech discourse

Peer reviewed

Beat that Word : How Listeners Integrate Beat Gesture and Focus in Multimodal Speech Discourse

Peer reviewed

Work at hand : an exploration of gesture in the context of work and everyday life to inform the design of gestural input devices

This research explores gestures used in the context of activities in the workplace and in everyday life in order to understand requirements and devise concepts for the design of gestural information applicances. A collaborative method of video interaction analysis devised to suit design explorations, the Video Card Game, was used to capture and analyse how gesture is used in the context of six different domains: the dentist's office; PDA and mobile phone use; the experimental biologist's laboratory; a city ferry service; a video cassette player repair shop; and a factory flowmeter assembly station. Findings are presented in the form of gestural themes, derived from the tradition of qualitative analysis but bearing some similarity to Alexandrian patterns. Implications for the design of gestural devices are discussed.

Movements in gesture interfaces

There is a mismatch between the kinds of movements used in gesture interfaces and our existing theoretical understandings of gesture. We need to re-examine the assumptions of gesture research and develop theory more suited to gesture interface design. In addition to improved theory, we need to develop ways for participants in the process of design to adapt, extend and develop theory for their own design contexts. Gesture interface designers should approach theory as a contingent resource for design actions that is responsive to the needs of the design process.

Primary students’ interpretation of maps : gesture use and mapping knowledge

Maps are used to represent three-dimensional space and are integral to a range of everyday experiences. They are increasingly used in mathematics, being prominent both in school curricula and as a form of assessing students understanding of mathematics ideas. In order to successfully interpret maps, students need to be able to understand that maps: represent space, have their own perspective and scale, and their own set of symbols and texts. Despite the fact that maps have an increased prevalence in society and school, there is evidence to suggest that students have difficulty interpreting maps. This study investigated 43 primary-aged students’ (aged 9-12 years) verbal and gestural behaviours as they engaged with and solved map tasks. Within a multiliteracies framework that focuses on spatial, visual, linguistic, and gestural elements, the study investigated how students interpret map tasks. Specifically, the study sought to understand students’ skills and approaches used to solving map tasks and the gestural behaviours they utilised as they engaged with map tasks. The investigation was undertaken using the Knowledge Discovery in Data (KDD) design. The design of this study capitalised on existing research data to carry out a more detailed analysis of students’ interpretation of map tasks. Video data from an existing data set was reorganised according to two distinct episodes—Task Solution and Task Explanation—and analysed within the multiliteracies framework. Content Analysis was used with these data and through anticipatory data reduction techniques, patterns of behaviour were identified in relation to each specific map task by looking at task solution, task correctness and gesture use. The findings of this study revealed that students had a relatively sound understanding of general mapping knowledge such as identifying landmarks, using keys, compass points and coordinates. However, their understanding of mathematical concepts pertinent to map tasks including location, direction, and movement were less developed. Successful students were able to interpret the map tasks and apply relevant mathematical understanding to navigate the spatial demands of the map tasks while the unsuccessful students were only able to interpret and understand basic map conventions. In terms of their gesture use, the more difficult the task, the more likely students were to exhibit gestural behaviours to solve the task. The most common form of gestural behaviour was deictic, that is a pointing gesture. Deictic gestures not only aided the students capacity to explain how they solved the map tasks but they were also a tool which assisted them to navigate and monitor their spatial movements when solving the tasks. There were a number of implications for theory, learning and teaching, and test and curriculum design arising from the study. From a theoretical perspective, the findings of the study suggest that gesturing is an important element of multimodal engagement in mapping tasks. In terms of teaching and learning, implications include the need for students to utilise gesturing techniques when first faced with new or novel map tasks. As students become more proficient in solving such tasks, they should be encouraged to move beyond a reliance on such gesture use in order to progress to more sophisticated understandings of map tasks. Additionally, teachers need to provide students with opportunities to interpret and attend to multiple modes of information when interpreting map tasks.

Phenomenological modeling of cell-to-cell and beat-to-beat variability in isolated Guinea Pig ventricular myocytes

Experimental action potential (AP) recordings in isolated ventricular myoctes display significant temporal beat-to-beat variability in morphology and duration. Furthermore, significant cell-to-cell differences in AP also exist even for isolated cells originating from the same region of the same heart. However, current mathematical models of ventricular AP fail to replicate the temporal and cell-to-cell variability in AP observed experimentally. In this study, we propose a novel mathematical framework for the development of phenomenological AP models capable of capturing cell-to-cell and temporal variabilty in cardiac APs. A novel stochastic phenomenological model of the AP is developed, based on the deterministic Bueno-Orovio/Fentonmodel. Experimental recordings of AP are fit to the model to produce AP models of individual cells from the apex and the base of the guinea-pig ventricles. Our results show that the phenomenological model is able to capture the considerable differences in AP recorded from isolated cells originating from the location. We demonstrate the closeness of fit to the available experimental data which may be achieved using a phenomenological model, and also demonstrate the ability of the stochastic form of the model to capture the observed beat-to-beat variablity in action potential duration.

Detecting gesture force peaks for intuitive interaction

With the release of the Nintendo Wii in 2006, the use of haptic force gestures has become a very popular form of input for interactive entertainment. However, current gesture recognition techniques utilised in Nintendo Wii games fall prey to a lack of control when it comes to recognising simple gestures. This paper presents a simple gesture recognition technique called Peak Testing which gives greater control over gesture interaction. This recognition technique locates force peaks in continuous force data (provided by a gesture device such as the Wiimote) and then cancels any peaks which are not meant for input. Peak Testing is therefore technically able to identify movements in any direction. This paper applies this recognition technique to control virtual instruments and investigates how users respond to this interaction. The technique is then explored as the basis for a robust way to navigate menus with a simple flick of the wrist. We propose that this flick-form of interaction could be a very intuitive way to navigate Nintendo Wii menus instead of the current pointer techniques implemented.

Beat phenomena in metal nanowires, and their implications for resonance-based elastic property measurements

The elastic properties of 1D nanostructures such as nanowires are often measured experimentally through actuation of the nanowire at its resonance frequency, and then relating the resonance frequency to the elastic stiffness using elementary beam theory. In the present work, we utilize large scale molecular dynamics simulations to report a novel beat phenomenon in [110]oriented Ag nanowires. The beat phenomenon is found to arise from the asymmetry of the lattice spacing in the orthogonal elementary directions of the [110] nanowire, i.e. the [-110] and [001] directions, which results in two different principal moments of inertia. Because of this, actuations imposed along any other direction are found to decompose into two orthogonal vibrational components based on the actuation angle relative to these two elementary directions, with this phenomenon being generalizable to <110> FCC nanowires of different materials (Cu, Au, Ni, Pd and Pt). The beat phenomenon is explained using a discrete moment of inertia model based on the hard sphere assumption, the model is utilized to show that surface effects enhance the beat phenomenon, while the effect is reduced with increasing nanowires cross-sectional size or aspect ratio. Most importantly, due to the existence of the beat phenomena, we demonstrate that in resonance experiments only a single frequency component is expected to be observed, particularly when the damping ratio is relatively large or very small. Furthermore, for a large range of actuation angles, the lower frequency is more likely to be detected than the higher one, which implies that experimental predictions of Young’s modulus obtained from resonance may in fact be under predictions. The present study therefore has significant implications for experimental interpretations of Young’s modulus as obtained via resonance testing.

Getting the point : the role of gesture in managing intersubjectivity in a design activity

This paper illustrates the complexity of pointing as it is employed in a design workshop. Using the method of interaction analysis, we argue that pointing is not merely employed to index, locate, or fix reference to an object. It also constitutes a practice for reestablishing intersubjectivity and solving interactional trouble such as misunderstandings or disagreements by virtue of enlisting something as part of the participants’ shared experience. We use this analysis to discuss implications for how such practices might be supported with computer mediation, arguing for a “bricolage” approach to systems development that emphasizes the provision of resources for users to collaboratively negotiate the accomplishment of intersubjectivity ra- ther than systems that try to support pointing as a specific gestural action.

Framing Movements for Gesture Interface Design

Gesture interfaces are an attractive avenue for human-computer interaction, given the range of expression that people are able to engage when gesturing. Consequently, there is a long running stream of research into gesture as a means of interaction in the field of human-computer interaction. However, most of this research has focussed on the technical challenges of detecting and responding to people’s movements, or on exploring the interaction possibilities opened up by technical developments. There has been relatively little research on how to actually design gesture interfaces, or on the kinds of understandings of gesture that might be most useful to gesture interface designers. Running parallel to research in gesture interfaces, there is a body of research into human gesture, which would seem a useful source to draw knowledge that could inform gesture interface design. However, there is a gap between the ways that ‘gesture’ is conceived of in gesture interface research compared to gesture research. In this dissertation, I explore this gap and reflect on the appropriateness of existing research into human gesturing for the needs of gesture interface design. Through a participatory design process, I designed, prototyped and evaluated a gesture interface for the work of the dental examination. Against this grounding experience, I undertook an analysis of the work of the dental examination with particular focus on the roles that gestures play in the work to compare and discuss existing gesture research. I take the work of the gesture researcher McNeill as a point of focus, because he is widely cited within gesture interface research literature. I show that although McNeill’s research into human gesture can be applied to some important aspects of the gestures of dentistry, there remain range of gestures that McNeill’s work does not deal with directly, yet which play an important role in the work and could usefully be responded to with gesture interface technologies. I discuss some other strands of gesture research, which are less widely cited within gesture interface research, but offer a broader conception of gesture that would be useful for gesture interface design. Ultimately, I argue that the gap in conceptions of gesture between gesture interface research and gesture research is an outcome of the different interests that each community brings to bear on the research. What gesture interface research requires is attention to the problems of designing gesture interfaces for authentic context of use and assessment of existing theory in light of this.

Beat-mixing Rock music: Rock and electronic dance music merge to create the Manarays

This research introduces the proposition that Electronic Dance Music’s beat-mixing function could be implemented to create immediacy in other musical genres. The inclusion of rhythmic sections at the beginning and end of each musical work created a ‘DJ friendly’ environment. The term used in this thesis to refer to the application of beat-mixing in Rock music is ‘ClubRock’. Collaboration between a number of DJs and Rock music professionals applied the process of beat-mixing to blend Rock tracks to produce a continuous ClubRock set. The DJ technique of beat-mixing Rock music transformed static renditions into a fluid creative work. The hybridisation of the two genres, EDM and Rock, resulted in a contribution to Rock music compositional approaches and the production of a unique Rock album; Manarays—Get Lucky.

Evoking gesture in interactive art

In this paper, we describe an interactive artwork that uses large body gestures as its primary interactive mode. The artist intends the work to provoke active reflection in the audience by way of gesture and content. The technology is not the focus, rather the aim is to provoke memory, to elicit feelings of connective human experiences in a required-to-participate audience. We find the work provokes a diverse and contradictory set of responses. The methods used to understand this include qualitative methods common to evaluating interactive art works, as well as in-depth discussions with the artist herself. This paper is relevant to the Human - Centered Computing track because in all stages of the design of the work - as well as the evaluation - the focus is on the human aspect; the computing is designed to enable all-too-human responses.