A multimodal and adaptive real-time human -computer interface for peoples with severe motor disability

This dissertation introduces the design of a multimodal, adaptive real-time assistive system as an alternate human computer interface that can be used by individuals with severe motor disabilities. The proposed design is based on the integration of a remote eye-gaze tracking system, voice recognition software, and a virtual keyboard. The methodology relies on a user profile that customizes eye gaze tracking using neural networks. The user profiling feature facilitates the notion of universal access to computing resources for a wide range of applications such as web browsing, email, word processing and editing. ^ The study is significant in terms of the integration of key algorithms to yield an adaptable and multimodal interface. The contributions of this dissertation stem from the following accomplishments: (a) establishment of the data transport mechanism between the eye-gaze system and the host computer yielding to a significantly low failure rate of 0.9%; (b) accurate translation of eye data into cursor movement through congregate steps which conclude with calibrated cursor coordinates using an improved conversion function; resulting in an average reduction of 70% of the disparity between the point of gaze and the actual position of the mouse cursor, compared with initial findings; (c) use of both a moving average and a trained neural network in order to minimize the jitter of the mouse cursor, which yield an average jittering reduction of 35%; (d) introduction of a new mathematical methodology to measure the degree of jittering of the mouse trajectory; (e) embedding an onscreen keyboard to facilitate text entry, and a graphical interface that is used to generate user profiles for system adaptability. ^ The adaptability nature of the interface is achieved through the establishment of user profiles, which may contain the jittering and voice characteristics of a particular user as well as a customized list of the most commonly used words ordered according to the user's preferences: in alphabetical or statistical order. This allows the system to successfully provide the capability of interacting with a computer. Every time any of the sub-system is retrained, the accuracy of the interface response improves even more. ^

(Re)criando (mini)contos de terror: uma experiência de ensino de leitura e escrita de gênero multimodal

Given the multiplicity of languages and media present in contemporary texts, the work with digital genres characterizes itself as essential for teaching, reading and writing. Virtual media is already present in many dayly activities that require the use of language. This shows that the globalized world brings new demands of literacy and various reading practices. Given this perspective, we propose to work the multiliteracies present in the new texts from the enunciative discourse Bakhtinian assumptions. For this, we chose to be as the object of research / intervention, the horror flash fiction multimodal discursive genre, because it is a multissemiótico digital statement of virtual circulation. In this context, this study aimed to understand how the teaching of this kind can contribute to the development of knowledge related to reading and text production, required by multiliteracies, by performing a Didactic Sequence in the classroom, specifically for two classes of elementary school, 7th and 8th grades of public school. The research was based on Bakhtin's theory and the Circle (2009, 2011) on gender perspective in a dialogic and the proposed Dolz and Schneuwly (2004) for the text of teaching through sequences Teaching. We also use the precepts of multiliteracies focused on Rojo (2012, 2013). The methodology used was based on a qualitative approach. We consider the analysis of minicontos produced by students, it's own hibridism of multiliteracies, the discursive characteristics such as composition, style and subject content, in addition to relations dialogicity present in these statements. At the end of the study, we realized that our intervention contributed to the expansion of knowledge of the subjects involved related to reading and multimodal genre production.

A probabilistic analysis of the biometrics menagerie existence: case study in fingerprint data

Until recently the use of biometrics was restricted to high-security environments and criminal identification applications, for economic and technological reasons. However, in recent years, biometric authentication has become part of daily lives of people. The large scale use of biometrics has shown that users within the system may have different degrees of accuracy. Some people may have trouble authenticating, while others may be particularly vulnerable to imitation. Recent studies have investigated and identified these types of users, giving them the names of animals: Sheep, Goats, Lambs, Wolves, Doves, Chameleons, Worms and Phantoms. The aim of this study is to evaluate the existence of these users types in a database of fingerprints and propose a new way of investigating them, based on the performance of verification between subjects samples. Once introduced some basic concepts in biometrics and fingerprint, we present the biometric menagerie and how to evaluate them.

A probabilistic analysis of the biometrics menagerie existence: case study in fingerprint data

Until recently the use of biometrics was restricted to high-security environments and criminal identification applications, for economic and technological reasons. However, in recent years, biometric authentication has become part of daily lives of people. The large scale use of biometrics has shown that users within the system may have different degrees of accuracy. Some people may have trouble authenticating, while others may be particularly vulnerable to imitation. Recent studies have investigated and identified these types of users, giving them the names of animals: Sheep, Goats, Lambs, Wolves, Doves, Chameleons, Worms and Phantoms. The aim of this study is to evaluate the existence of these users types in a database of fingerprints and propose a new way of investigating them, based on the performance of verification between subjects samples. Once introduced some basic concepts in biometrics and fingerprint, we present the biometric menagerie and how to evaluate them.

Beat that word : How listeners integrate beat gesture and focus in multimodal speech discourse

Peer reviewed

Beat that Word : How Listeners Integrate Beat Gesture and Focus in Multimodal Speech Discourse

Peer reviewed

The Material Dynamics of a London-French Blog: A Multimodal Reading of Migrant Habitus

Through a fine-grained reading of a London-French blog, this article aims to shed light on the lived experience of the French community in London. The ethnosemiotic conceptual framework brings together ethnographic and semiotic schools of thought, focusing in particular on Pierre Bourdieu’s concept of habitus and Gunther Kress’s multimodal social semiotic analytical model. Habitus is broken down into its material manifestations of habitat, habit and habituation, all displayed in the blog and revealing of the blogger’s identity and positioning within the migration setting. As all modes are considered to be of equal semiotic potential, equivalent emphasis is placed on the multiple modes of meaning-making present in the blog, such as layout, colour, typography and language. By examining the dynamic relationships between blogger and audience, subjectivity and objectivity, on-line and on-land habitus, and intermodal dynamics themselves, through the prism of multimodality, hidden facets of the blogger’s cultural identity and sense of community belonging within the diasporic context begin to materialise.

Multimodal Education for Very Young Learners

Invited Plenary Speaker

Deep Head Pose: Gaze-Direction Estimation in Multimodal Video

In this paper we present a convolutional neuralnetwork (CNN)-based model for human head pose estimation inlow-resolution multi-modal RGB-D data. We pose the problemas one of classification of human gazing direction. We furtherfine-tune a regressor based on the learned deep classifier. Next wecombine the two models (classification and regression) to estimateapproximate regression confidence. We present state-of-the-artresults in datasets that span the range of high-resolution humanrobot interaction (close up faces plus depth information) data tochallenging low resolution outdoor surveillance data. We buildupon our robust head-pose estimation and further introduce anew visual attention model to recover interaction with theenvironment. Using this probabilistic model, we show thatmany higher level scene understanding like human-human/sceneinteraction detection can be achieved. Our solution runs inreal-time on commercial hardware

Multimodal attention system for an interactive robot

[EN]Social robots are receiving much interest in the robotics community. The most important goal for such robots lies in their interaction capabilities. An attention system is crucial, both as a filter to center the robot’s perceptual resources and as a mean of letting the observer know that the robot has intentionality. In this paper a simple but flexible and functional attentional model is described. The model, which has been implemented in an interactive robot currently under development, fuses both visual and auditive information extracted from the robot’s environment, and can incorporate knowledge-based influences on attention.

Clinton Multimodal (U. S. 30 Mississippi River Bridge) Study, 2003

The purpose of this memorandum is to document the benefit-cost analysis of the river crossing concept alternatives described in the "Concept Alternatives Technical Memo." Benefit-cost studies are designed to measure, in dollars, the potential positive or negative impacts of large-scale construction projects. The concept alternatives analyzed include improvements to the Union Pacific Railroad (UPRR) River Crossing and the U.S. Highway 30 (U.S. 30) River Crossing.

La explotación de las artes en la promoción de la Marca Finlandia: análisis crítico del discurso multimodal

En este trabajo de fin de máster se investiga qué impacto tuvo la Marca Finlandia en el uso de las artes finlandesas en Focus – economía y tecnología, una revista publicada por el Ministerio de Asuntos Exteriores de Finlandia durante los años más profundos de la crisis económica 2008-2013. En 2008 el Ministro de Asuntos Exteriores de entonces, Alexander Stubb, designó una delegación para desarrollar la marca país de Finlandia, fruto de un proceso del que se presentó en 2010 un documento llamado Tehtävä Suomelle! (¡Misión para Finlandia!), cuyo objetivo fue concretar el diseño y las funciones de la Marca Finlandia. Este trabajo se enfoca en la utilización del arte y de artistas finlandeses en el material creado para apoyar la promoción de la imagen del país y, en última instancia, su competividad económica. Para tener en cuenta el contexto del material objeto de estudio, presentamos el concepto de la marca país y distintos planteamientos teóricos para su realización. Siendo una estrategia estatal polémica, recorremos los puntos de crítica más importantes sobre esta práctica promocional. También nos familiarizamos con la historia de la promoción nacional antes de introducir el presente proyecto, la Marca Finlandia, y sus objetivos. En la parte teórica se presentan los conceptos relevantes para el estudio: el discurso y su análisis crítico, enfatizando su capacidad ideológica para mantener y crear relaciones del poder muy a menudo desiguales. Ya que nuestro material de estudio empírico está compuesto de publicaciones físicas con dimensiones textuales, visuales y hápticas, nuestro método de investigación es observar críticamente estas relaciones desde la perspectiva multimodal. En la parte empírica del trabajo analizamos la composición de la comunicación multimodal del material imprimido, producido por el Ministerio de Asuntos Exteriores de Finlandia y sus socios de cooperación. Las revistas anuales Focus– Economía y tecnología se enfocan en noticias y artículos sobre los sectores de la economía y tecnología, pero también incluyen contenido sobre el diseño, la música, la moda, la arquitectura y la danza finlandeses. En nuestro análisis recorremos los campos verbales y visuales utilizados en las revistas para investigar de qué manera se presentaban las artes finlandesas en la operación de Marca Finlandia. Detectamos representaciones textuales, resaltaciones visuales y la combinación de ambos, lo que servía a las metas preterminadas y la imagen requerida por la delegación diseñadora de la marca país. Las revistas compartían el discurso en común que corría paralelo con los objetivos de la Marca Finlandia, que a su vez se puede ver como parte del discurso hegemónico neoliberal en general.

Interaction techniques with novel multimodal feedback for addressing gesture-sensing systems

Users need to be able to address in-air gesture systems, which means finding where to perform gestures and how to direct them towards the intended system. This is necessary for input to be sensed correctly and without unintentionally affecting other systems. This thesis investigates novel interaction techniques which allow users to address gesture systems properly, helping them find where and how to gesture. It also investigates audio, tactile and interactive light displays for multimodal gesture feedback; these can be used by gesture systems with limited output capabilities (like mobile phones and small household controls), allowing the interaction techniques to be used by a variety of device types. It investigates tactile and interactive light displays in greater detail, as these are not as well understood as audio displays. Experiments 1 and 2 explored tactile feedback for gesture systems, comparing an ultrasound haptic display to wearable tactile displays at different body locations and investigating feedback designs. These experiments found that tactile feedback improves the user experience of gesturing by reassuring users that their movements are being sensed. Experiment 3 investigated interactive light displays for gesture systems, finding this novel display type effective for giving feedback and presenting information. It also found that interactive light feedback is enhanced by audio and tactile feedback. These feedback modalities were then used alongside audio feedback in two interaction techniques for addressing gesture systems: sensor strength feedback and rhythmic gestures. Sensor strength feedback is multimodal feedback that tells users how well they can be sensed, encouraging them to find where to gesture through active exploration. Experiment 4 found that they can do this with 51mm accuracy, with combinations of audio and interactive light feedback leading to the best performance. Rhythmic gestures are continuously repeated gesture movements which can be used to direct input. Experiment 5 investigated the usability of this technique, finding that users can match rhythmic gestures well and with ease. Finally, these interaction techniques were combined, resulting in a new single interaction for addressing gesture systems. Using this interaction, users could direct their input with rhythmic gestures while using the sensor strength feedback to find a good location for addressing the system. Experiment 6 studied the effectiveness and usability of this technique, as well as the design space for combining the two types of feedback. It found that this interaction was successful, with users matching 99.9% of rhythmic gestures, with 80mm accuracy from target points. The findings show that gesture systems could successfully use this interaction technique to allow users to address them. Novel design recommendations for using rhythmic gestures and sensor strength feedback were created, informed by the experiment findings.

This Is a Gorgeous-looking Movie”: Multimodal Interplay between Commentary and Subject Materials in Movie Review Entertainment

The present thesis is a study of movie review entertainment (MRE) which is a contemporary Internet-based genre of texts. MRE are movie reviews in video form which are published online, usually as episodes of an MRE web show. Characteristic to MRE is combining humor and honest opinions in varying degrees as well as the use of subject materials, i.e. clips of the movies, as a part of the review. The study approached MRE from a linguistic perspective aiming to discover 1) whether MRE is primarily text- or image-based and what the primary functions of the modes are, 2) how a reviewer linguistically combines subject footage to her/his commentary?, 3) whether there is any internal variation in MRE regarding the aforementioned questions, and 4) how suitable the selected models and theories are in the analysis of this type of contemporary multimodal data. To answer the aforementioned questions, the multimodal system of image—text relations by Martinec and Salway (2005) in combination with categories of cohesion by Halliday and Hasan (1976) were applied to four full MRE videos which were transcribed in their entirety for the study. The primary data represent varying types of MRE: a current movie review, an analytic essay, a riff review, and a humorous essay. The results demonstrated that image vs. text prioritization can vary between reviews and also within a review. The current movie review and the two essays were primarily commentary-focused whereas the riff review was significantly more dependent on the use of imagery as the clips are a major source of humor which is a prominent value in that type of a review. In addition to humor, clips are used to exemplify the commentary. A reviewer also relates new information to the imagery as well as uses two modes to present the information in a review. Linguistically, the most frequent case was that the reviewer names participants and processes lexically in the commentary. Grammatical relations (reference items such as pronouns and adverbs and conjunctive items in the riff review) were also encountered. There was internal variation to a considerable degree. The methods chosen were deemed appropriate to answer the research questions. Further study could go beyond linguistics to include, for instance, genre and media studies.