Interaction techniques with novel multimodal feedback for addressing gesture-sensing systems

Users need to be able to address in-air gesture systems, which means finding where to perform gestures and how to direct them towards the intended system. This is necessary for input to be sensed correctly and without unintentionally affecting other systems. This thesis investigates novel interaction techniques which allow users to address gesture systems properly, helping them find where and how to gesture. It also investigates audio, tactile and interactive light displays for multimodal gesture feedback; these can be used by gesture systems with limited output capabilities (like mobile phones and small household controls), allowing the interaction techniques to be used by a variety of device types. It investigates tactile and interactive light displays in greater detail, as these are not as well understood as audio displays. Experiments 1 and 2 explored tactile feedback for gesture systems, comparing an ultrasound haptic display to wearable tactile displays at different body locations and investigating feedback designs. These experiments found that tactile feedback improves the user experience of gesturing by reassuring users that their movements are being sensed. Experiment 3 investigated interactive light displays for gesture systems, finding this novel display type effective for giving feedback and presenting information. It also found that interactive light feedback is enhanced by audio and tactile feedback. These feedback modalities were then used alongside audio feedback in two interaction techniques for addressing gesture systems: sensor strength feedback and rhythmic gestures. Sensor strength feedback is multimodal feedback that tells users how well they can be sensed, encouraging them to find where to gesture through active exploration. Experiment 4 found that they can do this with 51mm accuracy, with combinations of audio and interactive light feedback leading to the best performance. Rhythmic gestures are continuously repeated gesture movements which can be used to direct input. Experiment 5 investigated the usability of this technique, finding that users can match rhythmic gestures well and with ease. Finally, these interaction techniques were combined, resulting in a new single interaction for addressing gesture systems. Using this interaction, users could direct their input with rhythmic gestures while using the sensor strength feedback to find a good location for addressing the system. Experiment 6 studied the effectiveness and usability of this technique, as well as the design space for combining the two types of feedback. It found that this interaction was successful, with users matching 99.9% of rhythmic gestures, with 80mm accuracy from target points. The findings show that gesture systems could successfully use this interaction technique to allow users to address them. Novel design recommendations for using rhythmic gestures and sensor strength feedback were created, informed by the experiment findings.

Effects of modality, urgency and situation on responses to multimodal warnings for drivers

Signifying road-related events with warnings can be highly beneficial, especially when imminent attention is needed. This thesis describes how modality, urgency and situation can influence driver responses to multimodal displays used as warnings. These displays utilise all combinations of audio, visual and tactile modalities, reflecting different urgency levels. In this way, a new rich set of cues is designed, conveying information multimodally, to enhance reactions during driving, which is a highly visual task. The importance of the signified events to driving is reflected in the warnings, and safety-critical or non-critical situations are communicated through the cues. Novel warning designs are considered, using both abstract displays, with no semantic association to the signified event, and language-based ones, using speech. These two cue designs are compared, to discover their strengths and weaknesses as car alerts. The situations in which the new cues are delivered are varied, by simulating both critical and non-critical events and both manual and autonomous car scenarios. A novel set of guidelines for using multimodal driver displays is finally provided, considering the modalities utilised, the urgency signified, and the situation simulated.

Gestural communication of music structure during solo classical piano performance

The production and perception of music is a multimodal activity involving auditory, visual and conceptual processing, integrating these with prior knowledge and environmental experience. Musicians utilise expressive physical nuances to highlight salient features of the score. The question arises within the literature as to whether performers’ non-technical, non-sound-producing movements may be communicatively meaningful and convey important structural information to audience members and co-performers. In the light of previous performance research (Vines et al., 2006, Wanderley, 2002, Davidson, 1993), and considering findings within co-speech gestural research and auditory and audio-visual neuroscience, this thesis examines the nature of those movements not directly necessary for the production of sound, and their particular influence on audience perception. Within the current research 3D performance analysis is conducted using the Vicon 12- camera system and Nexus data-processing software. Performance gestures are identified as repeated patterns of motion relating to music structure, which not only express phrasing and structural hierarchy but are consistently and accurately interpreted as such by a perceiving audience. Gestural characteristics are analysed across performers and performance style using two Chopin preludes selected for their diverse yet comparable structures (Opus 28:7 and 6). Effects on perceptual judgements of presentation modes (visual-only, auditory-only, audiovisual, full- and point-light) and viewing conditions are explored. This thesis argues that while performance style is highly idiosyncratic, piano performers reliably generate structural gestures through repeated patterns of upper-body movement. The shapes and locations of phrasing motions are identified particular to the sample of performers investigated. Findings demonstrate that despite the personalised nature of the gestures, performers use increased velocity of movements to emphasise musical structure and that observers accurately and consistently locate phrasing junctures where these patterns and variation in motion magnitude, shape and velocity occur. By viewing performance motions in polar (spherical) rather than cartesian coordinate space it is possible to get mathematically closer to the movement generated by each of the nine performers, revealing distinct patterns of motion relating to phrasing structures, regardless of intended performance style. These patterns are highly individualised both to each performer and performed piece. Instantaneous velocity analysis indicates a right-directed bias of performance motion variation at salient structural features within individual performances. Perceptual analyses demonstrate that audience members are able to accurately and effectively detect phrasing structure from performance motion alone. This ability persists even for degraded point-light performances, where all extraneous environmental information has been removed. The relative contributions of audio, visual and audiovisual judgements demonstrate that the visual component of a performance does positively impact on the over- all accuracy of phrasing judgements, indicating that receivers are most effective in their recognition of structural segmentations when they can both see and hear a performance. Observers appear to make use of a rapid online judgement heuristics, adjusting response processes quickly to adapt and perform accurately across multiple modes of presentation and performance style. In line with existent theories within the literature, it is proposed that this processing ability may be related to cognitive and perceptual interpretation of syntax within gestural communication during social interaction and speech. Findings of this research may have future impact on performance pedagogy, computational analysis and performance research, as well as potentially influencing future investigations of the cognitive aspects of musical and gestural understanding.

Russian social networks on the web: cohesion and coherence in Vkontakte

In this thesis connections between messages on the public wall of the Russian social network Vkontakte are analysed and classified. A total of 1818 messages from three different Vkontakte groups were collected and analysed according to a new framework based on Halliday and Hasan’s (1976) research into cohesion and Simmons’s (1981) adaptation of their classification for Russian. The two categories of textuality, cohesion and coherence, describe the linguistic connections between messages. The main aim was to find out how far the traditional categories of cohesion are applicable to an online social network including written text as well as multimedia-files. In addition to linguistic cohesion the pragmatic and topic coherence between Vkontakte messages was also analysed. The analysis of pragmatic coherence classifies the messages with acts according to their pragmatic function in relation to surrounding messages. Topic coherence analyses the content of the messages, describes where a topic begins, changes or is abandoned. Linguistic cohesion, topic coherence and pragmatic coherence enable three different types of connections between messages and these together form a coherent communication on the message wall. The cohesion devices identified by Halliday and Hasan and Simmons were found to occur in these texts, but additional devices were also identified: these are multimodal, graphical and grammatical cohesion.

Subtitling Chinese cinema: a case study of Zhang Yimou’s films

In recent years, more and more Chinese films have been exported abroad. This thesis intends to explore the subtitling of Chinese cinema into English, with Zhang Yimou’s films as a case study. Zhang Yimou is arguably the most critically and internationally acclaimed Chinese filmmaker, who has experimented with a variety of genres of films. I argue that in the subtitling of his films, there is an obvious adoption of the domestication translation strategy that reduces or even omits Chinese cultural references. I try to discover what cultural categories or perspectives of China are prone to the domestication of translation and have formulated five categories: humour, politeness, dialect, history and songs and the Peking Opera. My methodology is that I compare the source Chinese dialogue lines with the existing English subtitles by providing literal translations of the source lines, and I will also give my alternative translations that tend to retain the source cultural references better. I also speculate that the domestication strategy is frequently employed by subtitlers possibly because the subtitlers assume the source cultural references are difficult for target language subtitle readers to comprehend, even if they are translated into a target language. However, subtitle readers are very likely to understand more than what the dialogue lines and the target language subtitles express, because films are multimodal entities and verbal information is not the only source of information for subtitle readers. The image and the sound are also significant sources of information for subtitle readers who are constantly involved in a dynamic film-watching experience. They are also expected to grasp visual and acoustic information. The complete omission or domestication of source cultural references might also affect their interpretation of the non-verbal cues. I also contemplate that the translation, which frequently domesticates the source culture carried out by a translator who is also a native speaker of the source language, is ‘submissive translation’.