Improving surgery operations by means of cloud systems and distributed user interfaces

Surgical interventions are usually performed in an operation room; however, access to the information by the medical team members during the intervention is limited. While in conversations with the medical staff, we observed that they attach significant importance to the improvement of the information and communication direct access by queries during the process in real time. It is due to the fact that the procedure is rather slow and there is lack of interaction with the systems in the operation room. These systems can be integrated on the Cloud adding new functionalities to the existing systems the medical expedients are processed. Therefore, such a communication system needs to be built upon the information and interaction access specifically designed and developed to aid the medical specialists. Copyright 2014 ACM.

El alineamiento modal en el preámbulo de reuniones por videoconferencia

En el presente artículo se introduce el concepto de alineamiento modal, un fenómeno interactivo característico del preámbulo de reuniones por videoconferencia, en las que la interacción puede llevarse a cabo a través del chat escrito, la imagen y la voz. Con tal propósito, se parte del modelo de interacción de Erving Goffman y la metodología del Análisis de la Conversación (AC). A través de una selección de ejemplos extraídos de un corpus de dieciocho interacciones por Adobe Connect 7.0, el análisis muestra que la selección del canal, dentro del contexto analizado, constituye un recurso para el alineamiento y la (re)configuración del marco de participación de las reuniones. Asimismo, se sugiere que dicho recurso es utilizado por los participantes como estrategia para gestionar la orientación recíproca y la toma de turno durante los preámbulos.

Multimodal automatic user disposition recognition in human-machine interaction

Magdeburg, Univ., Fak. für Elektrotechnik und Informationstechnik, Diss., 2013

Social network extraction and analysis based on multimodal dyadic interaction

Social interactions are a very important component in people"s lives. Social network analysis has become a common technique used to model and quantify the properties of social interactions. In this paper, we propose an integrated framework to explore the characteristics of a social network extracted from multimodal dyadic interactions. For our study, we used a set of videos belonging to New York Times" Blogging Heads opinion blog. The Social Network is represented as an oriented graph, whose directed links are determined by the Influence Model. The links" weights are a measure of the"influence" a person has over the other. The states of the Influence Model encode automatically extracted audio/visual features from our videos using state-of-the art algorithms. Our results are reported in terms of accuracy of audio/visual data fusion for speaker segmentation and centrality measures used to characterize the extracted social network.

Multimodal affective interaction: a comment on musical origins

THE RIGORS OF ESTABLISHING INNATENESS and domain specificity pose challenges to adaptationist models of music evolution. In articulating a series of constraints, the authors of the target articles provide strategies for investigating the potential origins of music. We propose additional approaches for exploring theories based on exaptation. We discuss a view of music as a multimodal system of engaging with affect, enabled by capacities of symbolism and a theory of mind.

Interaction techniques with novel multimodal feedback for addressing gesture-sensing systems

Users need to be able to address in-air gesture systems, which means finding where to perform gestures and how to direct them towards the intended system. This is necessary for input to be sensed correctly and without unintentionally affecting other systems. This thesis investigates novel interaction techniques which allow users to address gesture systems properly, helping them find where and how to gesture. It also investigates audio, tactile and interactive light displays for multimodal gesture feedback; these can be used by gesture systems with limited output capabilities (like mobile phones and small household controls), allowing the interaction techniques to be used by a variety of device types. It investigates tactile and interactive light displays in greater detail, as these are not as well understood as audio displays. Experiments 1 and 2 explored tactile feedback for gesture systems, comparing an ultrasound haptic display to wearable tactile displays at different body locations and investigating feedback designs. These experiments found that tactile feedback improves the user experience of gesturing by reassuring users that their movements are being sensed. Experiment 3 investigated interactive light displays for gesture systems, finding this novel display type effective for giving feedback and presenting information. It also found that interactive light feedback is enhanced by audio and tactile feedback. These feedback modalities were then used alongside audio feedback in two interaction techniques for addressing gesture systems: sensor strength feedback and rhythmic gestures. Sensor strength feedback is multimodal feedback that tells users how well they can be sensed, encouraging them to find where to gesture through active exploration. Experiment 4 found that they can do this with 51mm accuracy, with combinations of audio and interactive light feedback leading to the best performance. Rhythmic gestures are continuously repeated gesture movements which can be used to direct input. Experiment 5 investigated the usability of this technique, finding that users can match rhythmic gestures well and with ease. Finally, these interaction techniques were combined, resulting in a new single interaction for addressing gesture systems. Using this interaction, users could direct their input with rhythmic gestures while using the sensor strength feedback to find a good location for addressing the system. Experiment 6 studied the effectiveness and usability of this technique, as well as the design space for combining the two types of feedback. It found that this interaction was successful, with users matching 99.9% of rhythmic gestures, with 80mm accuracy from target points. The findings show that gesture systems could successfully use this interaction technique to allow users to address them. Novel design recommendations for using rhythmic gestures and sensor strength feedback were created, informed by the experiment findings.

Investigating interpreter-mediated interaction in the Chinese bilingual courtroom: Analysis based on a multimodal corpus

This research project is based on the Multimodal Corpus of Chinese Court Interpreting (MUCCCI [mutʃɪ]), a small-scale multimodal corpus on the basis of eight authentic court hearings with Chinese-English interpreting in Mainland China. The corpus has approximately 92,500 word tokens in total. Besides the transcription of linguistic and para-linguistic features, utilizing the facial expression classification rules suggested by Black and Yacoob (1995), MUCCCI also includes approximately 1,200 annotations of facial expressions linked to the six basic types of human emotions, namely, anger, disgust, happiness, surprise, sadness, and fear (Black & Yacoob, 1995). This thesis is an example of conducting qualitative analysis on interpreter-mediated courtroom interactions through a multimodal corpus. In particular, miscommunication events (MEs) and the reasons behind them were investigated in detail. During the analysis, although queries were conducted based on non-verbal annotations when searching for MEs, both verbal and non-verbal features were considered indispensable parts contributing to the entire context. This thesis also includes a detailed description of the compilation process of MUCCCI utilizing ELAN, from data collection to transcription, POS tagging and non-verbal annotation. The research aims at assessing the possibility and feasibility of conducting qualitative analysis through a multimodal corpus of court interpreting. The concept of integrating both verbal and non-verbal features to contribute to the entire context is emphasized. The qualitative analysis focusing on MEs can provide an inspiration for improving court interpreters’ performances. All the constraints and difficulties presented can be regarded as a reference for similar research in the future.

Multimodal Strategies in Development.Predictive Value of early simultaneosgesture-speech comination

The present study investigates the predictive value of the early appearance of simultaneous pointing-speech combinations. An experimental task was used to obtain a communicative productive sample from nineteen children at 1;0 and 1;3. Infant’s communicative productions, in combination with gaze joint engagement patterns, were analyzed in relation to different social conditions. The results show a significant effect of age and social condition on infants’ communicative productions. Gesture-speech combinations seem to work as a strong communicative resource to attract the adult’s attention in social demanding communicative contexts. Gaze joint engagement was used in combination with simultaneous pointing-speech combinations to attract adults’ attention during social demanding conditions. Finally, the use of simultaneous pointing-speech combinations at 1;0 in demanding conditions predicted greater expressive vocabulary acquisition at 1;3 and 1;6. These results indicate that the use of gesture-speech combinations may be considered a significant step towards the early integration of language components.

Acting Rehearsal in Collaborative Multimodal Mixed Reality Environments

This paper presents the use of our multimodal mixed reality telecommunication system to support remote acting rehearsal. The rehearsals involved two actors, located in London and Barcelona, and a director in another location in London. This triadic audiovisual telecommunication was performed in a spatial and multimodal collaborative mixed reality environment based on the 'destination-visitor' paradigm, which we define and put into use. We detail our heterogeneous system architecture, which spans the three distributed and technologically asymmetric sites, and features a range of capture, display, and transmission technologies. The actors' and director's experience of rehearsing a scene via the system are then discussed, exploring successes and failures of this heterogeneous form of telecollaboration. Overall, the common spatial frame of reference presented by the system to all parties was highly conducive to theatrical acting and directing, allowing blocking, gross gesture, and unambiguous instruction to be issued. The relative inexpressivity of the actors' embodiments was identified as the central limitation of the telecommunication, meaning that moments relying on performing and reacting to consequential facial expression and subtle gesture were less successful.

Acting rehearsal in collaborative multimodal mixed reality environments

This paper presents the use of our multimodal mixed reality telecommunication system to support remote acting rehearsal. The rehearsals involved two actors, located in London and Barcelona, and a director in another location in London. This triadic audiovisual telecommunication was performed in a spatial and multimodal collaborative mixed reality environment based on the 'destination-visitor' paradigm, which we define and put into use. We detail our heterogeneous system architecture, which spans the three distributed and technologically asymmetric sites, and features a range of capture, display, and transmission technologies. The actors' and director's experience of rehearsing a scene via the system are then discussed, exploring successes and failures of this heterogeneous form of telecollaboration. Overall, the common spatial frame of reference presented by the system to all parties was highly conducive to theatrical acting and directing, allowing blocking, gross gesture, and unambiguous instruction to be issued. The relative inexpressivity of the actors' embodiments was identified as the central limitation of the telecommunication, meaning that moments relying on performing and reacting to consequential facial expression and subtle gesture were less successful.

Multimodal counter-argumentation in the workplace : the contribution of gesture and gaze to the expression of disagreement

This paper examines argumentative talk-in-interaction in the workplace. It focuses on counter-argumentative references, which consist of the various resources that the opponent uses to refer to the origin/source of his/her opposition, namely the confronted position and the person who expressed it. Particular attention is paid to the relationship - in terms of sequential positioning and referential extension - between reported speech, polyphony, pointing gestures and shifts in gaze direction. Data are taken from workplace management meetings that have been recorded in New Zealand by the Language in the Workplace Project.

“IT’S ALL RIGHT”. Multimodal rightward spatial bias modified by age and praxis

The general goal of the present work was to study whether spatial perceptual asymmetry initially observed in linguistic dichotic listening studies is related to the linguistic nature of the stimuli and/or is modality-specific, as well as to investigate whether the spatial perceptual/attentional asymmetry changes as a function of age and sensory deficit via praxis. Several dichotic listening studies with linguistic stimuli have shown that the inherent perceptual right ear advantage (REA), which presumably results from the left lateralized linguistic functions (bottom-up processes), can be modified with executive functions (top-down control). Executive functions mature slowly during childhood, are well developed in adulthood, and decline as a function of ageing. In Study I, the purpose was to investigate with a cross-sectional experiment from a lifespan perspective the age-related changes in top-down control of REA for linguistic stimuli in dichotic listening with a forced-attention paradigm (DL). In Study II, the aim was to determine whether the REA is linguistic-stimulus-specific or not, and whether the lifespan changes in perceptual asymmetry observed in dichotic listening would exist also in auditory spatial attention tasks that put load on attentional control. In Study III, using visual spatial attention tasks, mimicking the auditory tasks applied in Study II, it was investigated whether or not the stimulus-non-specific rightward spatial bias found in auditory modality is a multimodal phenomenon. Finally, as it has been suggested that the absence of visual input in blind participants leads to improved auditory spatial perceptual and cognitive skills, the aim in Study IV was to determine, whether blindness modifies the ear advantage in DL. Altogether 180-190 right-handed participants between 5 and 79 years of age were studied in Studies I to III, and in Study IV the performance of 14 blind individuals was compared with that of 129 normally sighted individuals. The results showed that only rightward spatial bias was observed in tasks with intensive attentional load, independent of the type of stimuli (linguistic vs. non-linguistic) or the modality (auditory vs. visual). This multimodal rightward spatial bias probably results from a complex interaction of asymmetrical perceptual, attentional, and/or motor mechanisms. Most importantly, the strength of the rightward spatial bias changed as a function of age and augmented praxis due to sensory deficit. The efficiency of the performance in spatial attention tasks and the ability to overcome the rightward spatial bias increased during childhood, was at its best in young adulthood, and decreased as a function of ageing. Between the ages of 5 and 11 years probably at first develops movement and impulse control, followed by the gradual development of abilities to inhibit distractions and disengage attention. The errors especially in bilateral stimulus conditions suggest that a mild phenomenon resembling extinction can be observed throughout the lifespan, but especially the ability to distribute attention to multiple targets simultaneously decreases in the course of ageing. Blindness enhances the processing of auditory bilateral linguistic stimuli, the ability to overcome a stimulus-driven laterality effect related to speech sound perception, and the ability to direct attention to an appropriate spatial location. It was concluded that the ability to voluntarily suppress and inhibit the multimodal rightward spatial bias changes as a function of age and praxis due to sensory deficit and probably reflects the developmental level of executive functions.

Analyse de la traduction d’un texte multimodal : la bande dessinée : le cas de Mujeres alteradas

La présente recherche porte sur la traduction de la bande dessinée. Ce sujet, auparavant négligé par les traductologes, commence à susciter l’intérêt des chercheurs à partir les années 80. Toutefois, la plupart des travaux se sont concentrés sur l’aspect linguistique des BD. Ce mémoire, par contre, aborde la bande dessinée comme un texte multimodal. Il s’inscrit ainsi à la croisée des domaines de la traduction et de la multimodalité telle que proposée dans les travaux de Gunther Kress et Theo Van Leeuwen (2001). L’objectif de cette recherche est d’implanter un outil d’analyse pour la bande dessinée qui permettrait de rendre compte des différents modes intervenant dans le texte. Cet outil, conçu pour la présente recherche, a été développé à partir des travaux de Hatim et Mason (1990, 1997) sur les trois dimensions de la situation de communication : transaction communicative, action pragmatique et interaction sémiotique. L’analyse menée ici porte sur la traduction vers le français de la bande dessinée argentine Mujeres alteradas de Maitena Burundarena, parue sous le titre de Les déjantées.

Queue-based agent architecture for multimodal interfaces

This paper presents a queue-based agent architecture for multimodal interfaces. Using a novel approach to intelligently organise both agents and input data, this system has the potential to outperform current state-of-the-art multimodal systems, while at the same time allowing greater levels of interaction and flexibility. This assertion is supported by simulation test results showing that significant improvements can be obtained over normal sequential agent scheduling architectures. For real usage, this translates into faster, more comprehensive systems, without the limited application domain that restricts current implementations.