Improving Speech Interaction in Vehicles Using Context-Aware Information through A SCXML Framework

Speech Technologies can provide important benefits for the development of more usable and safe in-vehicle human-machine interactive systems (HMIs). However mainly due robustness issues, the use of spoken interaction can entail important distractions to the driver. In this challenging scenario, while speech technologies are evolving, further research is necessary to explore how they can be complemented with both other modalities (multimodality) and information from the increasing number of available sensors (context-awareness). The perceived quality of speech technologies can significantly be increased by implementing such policies, which simply try to make the best use of all the available resources; and the in vehicle scenario is an excellent test-bed for this kind of initiatives. In this contribution we propose an event-based HMI design framework which combines context modelling and multimodal interaction using a W3C XML language known as SCXML. SCXML provides a general process control mechanism that is being considered by W3C to improve both voice interaction (VoiceXML) and multimodal interaction (MMI). In our approach we try to anticipate and extend these initiatives presenting a flexible SCXML-based approach for the design of a wide range of multimodal context-aware HMI in-vehicle interfaces. The proposed framework for HMI design and specification has been implemented in an automotive OSGi service platform, and it is being used and tested in the Spanish research project MARTA for the development of several in-vehicle interactive applications.

Integración de Fuentes con Información de Contexto y Sistemas Interactivos

Durante el transcurso de esta Tesis Doctoral se ha realizado un estudio de la problemática asociada al desarrollo de sistemas de interacción hombre-máquina sensibles al contexto. Este problema se enmarca dentro de dos áreas de investigación: los sistemas interactivos y las fuentes de información contextual. Tradicionalmente la integración entre ambos campos se desarrollaba a través de soluciones verticales específicas, que abstraen a los sistemas interactivos de conocer los procedimientos de bajo nivel de acceso a la información contextual, pero limitan su interoperabilidad con otras aplicaciones y fuentes de información. Para solventar esta limitación se hace imprescindible potenciar soluciones interoperables que permitan acceder a la información del mundo real a través de procedimientos homogéneos. Esta problemática coincide perfectamente con los escenarios de \Computación Ubicua" e \Internet de las Cosas", donde se apunta a un futuro en el que los objetos que nos rodean serán capaces de obtener información del entorno y comunicarla a otros objetos y personas. Los sistemas interactivos, al ser capaces de obtener información de su entorno a través de la interacción con el usuario, pueden tomar un papel especial en este escenario tanto como consumidores como productores de información. En esta Tesis se ha abordado la integración de ambos campos teniendo en cuenta este escenario tecnológico. Para ello, en primer lugar se ha realizado un an álisis de las iniciativas más importantes para la definición y diseño de sistemas interactivos, y de las principales infraestructuras de suministro de información. Mediante este estudio se ha propuesto utilizar el lenguaje SCXML del W3C para el diseño de los sistemas interactivos y el procesamiento de los datos proporcionados por fuentes de contexto. Así, se ha reflejado cómo las capacidades del lenguaje SCXML para combinar información de diferentes modalidades pueden también utilizarse para procesar e integrar información contextual de diferentes fuentes heterogéneas, y por consiguiente diseñar sistemas de interacción sensibles al contexto. Del mismo modo se presenta a la iniciativa Sensor Web, y a su extensión semántica Semantic Sensor Web, como una iniciativa idónea para permitir un acceso y suministro homogéneo de la información a los sistemas interactivos sensibles al contexto. Posteriormente se han analizado los retos que plantea la integración de ambos tipos de iniciativas. Como resultado se ha conseguido establecer una serie de funcionalidades que son necesarias implementar para llevar a cabo esta integración. Utilizando tecnologías que aportan una gran flexibilidad al proceso de implementación y que se apoyan en recomendaciones y estándares actuales, se implementaron una serie de desarrollos experimentales que integraban las funcionalidades identificadas anteriormente. Finalmente, con el fin de validar nuestra propuesta, se realizaron un conjunto de experimentos sobre un entorno de experimentación que simula el escenario de la conducción. En este escenario un sistema interactivo se comunica con una extensión semántica de una plataforma basada en los estándares de la Sensor Web para poder obtener información y publicar las observaciones que el usuario realizaba al sistema. Los resultados obtenidos han demostrado la viabilidad de utilizar el lenguaje SCXML para el diseño de sistemas interactivos sensibles al contexto que requieren acceder a plataformas avanzadas de información para consumir y publicar información a la vez que interaccionan con el usuario. Del mismo modo, se ha demostrado cómo la utilización de tecnologías semánticas en los procesos de consulta y publicación de información puede facilitar la reutilización de la información publicada en infraestructuras Sensor Web por cualquier tipo de aplicación, y de este modo contribuir al futuro escenario de Internet de las Cosas. ABSTRACT In this Thesis, we have addressed the difficulties related to the development of context-aware human-machine interaction systems. This issue is part of two research fields: interactive systems and contextual information sources. Traditionally both fields have been integrated through domain-specific vertical solutions that allow interactive systems to access contextual information without having to deal with low-level procedures, but restricting their interoperability with other applications and heterogeneous data sources. Thus, it is essential to boost the research on interoperable solutions that provide access to real world information through homogeneous procedures. This issue perfectly matches with the scenarios of \Ubiquitous Computing" and \Internet of Things", which point toward a future in which many objects around us will be able to acquire meaningful information about the environment and communicate it to other objects and to people. Since interactive systems are able to get information from their environment through interaction with the user, they can play an important role in this scenario as they can both consume real-world data and produce enriched information. This Thesis deals with the integration of both fields considering this technological scenario. In order to do this, we first carried out an analysis of the most important initiatives for the definition and design of interactive systems, and the main infrastructures for providing information. Through this study the use of the W3C SCXML language is proposed for both the design of interactive systems and the processing of data provided by different context sources. Thus, this work has shown how the SCXML capabilities for combining information from different modalities can also be used to process and integrate contextual information from different heterogeneous sensor sources, and therefore to develope context-aware interaction systems. Similarly, we present the Sensor Web initiative, and its semantic extension Semantic Sensor Web, as an appropriate initiative to allow uniform access and delivery of information to the context-aware interactive systems. Subsequently we have analyzed the challenges of integrating both types of initiatives: SCXML and (Semantic) Sensor Web. As a result, we state a number of functionalities that are necessary to implement in order to perform this integration. By using technologies that provide exibility to the implementation process and are based on current recommendations and standards, we implemented a series of experimental developments that integrate the identified functionalities. Finally, in order to validate our approach, we conducted different experiments with a testing environment simulating a driving scenario. In this framework an interactive system can access a semantic extension of a Telco plataform, based on the standards of the Sensor Web, to acquire contextual information and publish observations that the user performed to the system. The results showed the feasibility of using the SCXML language for designing context-aware interactive systems that require access to advanced sensor platforms for consuming and publishing information while interacting with the user. In the same way, it was shown how the use of semantic technologies in the processes of querying and publication sensor data can assist in reusing and sharing the information published by any application in Sensor Web infrastructures, and thus contribute to realize the future scenario of \Internet of Things".

Public policies and multilingualism in HCI

Language is an essential aspect of human communication and interaction, not only between humans but also between humans and interactive systems. Indeed, the use of the language is a keystone for the design of interactive systems.Inappropriate use of language might limit the access of users to information and induce users to make mistakes or to lose control of interactive systems. Despite the existence of public policies, the diversity of languages poses serious problems when considering full-fledged/seamless support for all existing languages. And yet standardization processes have successfully defined mechanisms for ensuring cross-platform compatibility between languages, at least at the level of format and the set of characters that can be used.

Development of Interactive Materials by Engineering Students Using the Descartes Applets.

In a degree course such as Forestry Engineering, the general teaching objectives consist of explaining and helping students to understand the principles of Mechanics. For some time now we have encountered significant difficulties in teaching this subject due to the students' lack of motivation and to their insufficient prior preparation for the topic. If we add to this the discipline's inherent complexity and the students' preconceptions about the subject, these teaching difficulties become considerable. For this reason a series of didactic activities have been introduced sequentially in the teaching of this subject. This work describes the methodology, procedure and results for the action of developing a work project in groups using Descartes software. The results of this experiment can be considered very positive. Some of the critical preconceptions for learning the subject can be corrected, and the tutoring process in the classroom contributes to an improvement in teacherstudent communication. Since this scheme was established, the number of students taking part each academic year has increased, and this is the group with the greatest percentage of passing scores.

Experimental Prototype Merging Stereo Panoramic Video and Interactive 3D Content in a 5-sided CAVETM

Immersion and interaction have been identified as key factors influencing the quality of experience in stereoscopic video systems. An experimental prototype designed to explore the influence of these factors in 3D video applications is described here1. The focus is on the real-time insertion algorithm of new 3D models into the original video streams. Using this algorithm, our prototype is aimed to explore a new interaction paradigm ? similar to the augmented reality approach ? with 3D video applications.

Integration of Correlation Processes in Design Systems.

The integration of correlation processes in design systems has as a target measurements in 3D directly and according to the users criteria in order to generate the required database for the development of the project. In the phase of photogrammetric works, internal and external orientation parameters are calculated and stereo models are created from standard images. The aforementioned are integrated in the system where the measurement of the selected items is done by applying developed correlation algorithms. The processing period has the tools to carry out the calculations in an easy and automatic way, as well as image measurement techniques to acquire the most correct information. The proposed software development is done on Visual Studio platforms for PC, applying the most apt codes and symbols according to the terms of reference required for the design. The results of generating the data base in an interactive way with the geometric study of the structures, facilitates and improves the quality of the works in the projects.

An online e-Learning authoring tool to create interactive multi-device learning objects using e-Infrastructure resources

Education can take advantage of e-Infrastructures to provide teachers with new opportunities to increase students' motivation and engagement while they learn. Nevertheless, teachers need to find, integrate and customize the resources provided by e-Infrastructures in an easy way. This paper presents ViSH Editor, an innovative web-based e-Learning authoring tool that aims to allow teachers to create new learning objects using e-Infrastructure resources. These new learning objects are called Virtual Excursions and are created as reusable, granular and interoperable learning objects. This way they can be reused to build new ones and they can be integrated in websites or Learning Management Systems. Details about the design, development and the tool itself are explained in this paper as well as the concept, structure and metadata of the new learning objects. Lastly, some real examples of how to enrich learning using Virtual Excursions are exposed.

An interactive method of fuzzy probability elicitation in risk analysis

Expert knowledge is used to assign probabilities to events in many risk analysis models. However, experts sometimes find it hard to provide specific values for these probabilities, preferring to express vague or imprecise terms that are mapped using a previously defined fuzzy number scale. The rigidity of these scales generates bias in the probability elicitation process and does not allow experts to adequately express their probabilistic judgments. We present an interactive method for extracting a fuzzy number from experts that represents their probabilistic judgments for a given event, along with a quality measure of the probabilistic judgments, useful in a final information filtering and analysis sensitivity process.

Combining Text and Graphics for Interactive Exploration of Behavior Datasets

Modern sensor technologies and simulators applied to large and complex dynamic systems (such as road traffic networks, sets of river channels, etc.) produce large amounts of behavior data that are difficult for users to interpret and analyze. Software tools that generate presentations combining text and graphics can help users understand this data. In this paper we describe the results of our research on automatic multimedia presentation generation (including text, graphics, maps, images, etc.) for interactive exploration of behavior datasets. We designed a novel user interface that combines automatically generated text and graphical resources. We describe the general knowledge-based design of our presentation generation tool. We also present applications that we developed to validate the method, and a comparison with related work.

Facilitating the creation of interactive multi-device Learning Objects using an online authoring tool

Learning Objects facilitate reuse leading to cost and time savings as well as to the enhancement of the quality of educational resources. However, teachers find it difficult to create or to find high quality Learning Objects, and the ones they find need to be customized. Teachers can overcome this problem using suitable authoring systems that enable them to create high quality Learning Objects with little effort. This paper presents an open source online e-Learning authoring tool called ViSH Editor together with four novel interactive Learning Objects that can be created with it: Flashcards, Virtual Tours, Enriched Videos and Interactive Presentations. All these Learning Objects are created as web applications, which can be accessed via mobile devices. Besides, they can be exported to SCORM including their metadata in IEEE LOM format. All of them are described in the paper including an example of each. This approach for creating Learning Objects was validated through two evaluations: a survey among authors and a formal quality evaluation of 209 Learning Objects created with the tool. The results show that ViSH Editor facilitates educators the creation of high quality Learning Objects.

MedVir: an interactive representation system of multidimensional medical data applied to Traumatic Brain Injury's rehabilitation prediction

Clinicians could model the brain injury of a patient through his brain activity. However, how this model is defined and how it changes when the patient is recovering are questions yet unanswered. In this paper, the use of MedVir framework is proposed with the aim of answering these questions. Based on complex data mining techniques, this provides not only the differentiation between TBI patients and control subjects (with a 72% of accuracy using 0.632 Bootstrap validation), but also the ability to detect whether a patient may recover or not, and all of that in a quick and easy way through a visualization technique which allows interaction.

User experience with immersive and interactive media = Experiencia de usuario en entornos 3D inmersivos e interactivos

Estudios recientes promueven la integración de estímulos multisensoriales en activos multimedia con el fin de mejorar la experiencia de usuario mediante la estimulación de nuevos sentidos, más allá de la tradicional experiencia audiovisual. Del mismo modo, varios trabajos proponen la introducción de componentes de interacción capaces de complementar con nuevas características, funcionalidades y/o información la experiencia multimedia. Efectos sensoriales basados en el uso de nuevas técnicas de audio, olores, viento, vibraciones y control de la iluminación, han demostrado tener un impacto favorable en la sensación de Presencia, en el disfrute de la experiencia multimedia y en la calidad, relevancia y realismo de la misma percibidos por el usuario. Asimismo, los servicios basados en dos pantallas y la manipulación directa de (elementos en) la escena de video tienen el potencial de mejorar la comprensión, la concentración y la implicación proactiva del usuario en la experiencia multimedia. El deporte se encuentra entre los géneros con mayor potencial para integrar y explotar éstas soluciones tecnológicas. Trabajos previos han demostrado asimismo la viabilidad técnica de integrar éstas tecnologías con los estándares actualmente adoptados a lo largo de toda la cadena de transmisión de televisión. De este modo, los sistemas multimedia enriquecidos con efectos sensoriales, los servicios interactivos multiplataforma y un mayor control del usuario sobre la escena de vídeo emergen como nuevas formas de llevar la multimedia immersiva e interactiva al mercado de consumo de forma no disruptiva. Sin embargo, existen numerosas interrogantes relativas a los efectos sensoriales y/o soluciones interactivas más adecuadas para complementar un contenido audiovisual determinado o a la mejor manera de de integrar y combinar dichos componentes para mejorar la experiencia de usuario de un segmento de audiencia objetivo. Además, la evidencia científica sobre el impacto de factores humanos en la experiencia de usuario con estas nuevas formas de immersión e interacción en el contexto multimedia es aún insuficiente y en ocasiones, contradictoria. Así, el papel de éstos factores en el potencial de adopción de éstas tecnologías ha sido amplia-mente ignorado. La presente tesis analiza el impacto del audio binaural, efectos sensoriales (de iluminación y olfativos), interacción con objetos 3D integrados en la escena de vídeo e interacción con contenido adicional utilizando una segunda pantalla en la experiencia de usuario con contenidos de deporte. La posible influencia de dichos componentes en las variables dependientes se explora tanto a nivel global (efecto promedio) como en función de las características de los usuarios (efectos heterogéneos). Para ello, se ha llevado a cabo un experimento con usuarios orientado a explorar la influencia de éstos componentes immersivos e interactivos en dos grandes dimensiones de la experiencia multimedia: calidad y Presencia. La calidad de la experiencia multimedia se analiza en términos de las posibles variaciones asociadas a la calidad global y a la calidad del contenido, la imagen, el audio, los efectos sensoriales, la interacción con objetos 3D y la interacción con la segunda pantalla. El posible impacto en la Presencia considera dos de las dimensiones definidas por el cuestionario ITC-SOPI: Presencia Espacial (Spatial Presence) e Implicación (Engagement). Por último, los individuos son caracterizados teniendo en cuenta los siguientes atributos afectivos, cognitivos y conductuales: preferencias y hábitos en relación con el contenido, grado de conocimiento de las tecnologías integradas en el sistema, tendencia a involucrarse emocionalmente, tendencia a concentrarse en una actividad bloqueando estímulos externos y los cinco grandes rasgos de la personalidad: extroversión, amabilidad, responsabilidad, inestabilidad emocional y apertura a nuevas experiencias. A nivel global, nuestro estudio revela que los participantes prefieren el audio binaural frente al sistema estéreo y que los efectos sensoriales generan un aumento significativo del nivel de Presencia Espacial percibido por los usuarios. Además, las manipulaciones experimentales realizadas permitieron identificar una gran variedad de efectos heterogéneos. Un resultado interesante es que dichos efectos no se encuentran distribuidos de forma equitativa entre las medidas de calidad y Presencia. Nuestros datos revelan un impacto generalizado del audio binaural en la mayoría de las medidas de calidad y Presencia analizadas. En cambio, la influencia de los efectos sensoriales y de la interacción con la segunda pantalla se concentran en las medidas de Presencia y calidad, respectivamente. La magnitud de los efectos heterogéneos identificados está modulada por las siguientes características personales: preferencias en relación con el contenido, frecuencia con la que el usuario suele ver contenido similar, conocimiento de las tecnologías integradas en el demostrador, sexo, tendencia a involucrarse emocionalmente, tendencia a a concentrarse en una actividad bloqueando estímulos externos y niveles de amabilidad, responsabilidad y apertura a nuevas experiencias. Las características personales consideradas en nuestro experimento explicaron la mayor parte de la variación en las variables dependientes, confirmando así el importante (y frecuentemente ignorado) papel de las diferencias individuales en la experiencia multimedia. Entre las características de los usuarios con un impacto más generalizado se encuentran las preferencias en relación con el contenido, el grado de conocimiento de las tecnologías integradas en el sistema y la tendencia a involucrarse emocionalmente. En particular, los primeros dos factores parecen generar un conflicto de atención hacia el contenido versus las características/elementos técnicos del sistema, respectivamente. Asimismo, la experiencia multimedia de los fans del fútbol parece estar modulada por procesos emociona-les, mientras que para los no-fans predominan los procesos cognitivos, en particular aquellos directamente relacionados con la percepción de calidad. Abstract Recent studies encourage the integration of multi-sensorial stimuli into multimedia assets to enhance the user experience by stimulating other senses beyond sight and hearing. Similarly, the introduction of multi-modal interaction components complementing with new features, functionalities and/or information the multimedia experience is promoted. Sensory effects as odor, wind, vibration and light effects, as well as an enhanced audio quality, have been found to favour media enjoyment and to have a positive influence on the sense of Presence and on the perceived quality, relevance and reality of a multimedia experience. Two-screen services and a direct manipulation of (elements in) the video scene have the potential to enhance user comprehension, engagement and proactive involvement of/in the media experience. Sports is among the genres that could benefit the most from these solutions. Previous works have demonstrated the technical feasibility of implementing and deploying end-to-end solutions integrating these technologies into legacy systems. Thus, sensorially-enhanced media, two-screen services and an increased user control over the displayed scene emerge as means to deliver a new form of immersive and interactive media experiences to the mass market in a non-disruptive manner. However, many questions remain concerning issues as the specific interactive solutions or sensory effects that can better complement a given audiovisual content or the best way in which to integrate and combine them to enhance the user experience of a target audience segment. Furthermore, scientific evidence on the impact of human factors on the user experience with these new forms of immersive and interactive media is still insufficient and sometimes, contradictory. Thus, the role of these factors on the potential adoption of these technologies has been widely ignored. This thesis analyzes the impact of binaural audio, sensory (light and olfactory) effects, interaction with 3D objects integrated into the video scene and interaction with additional content using a second screen on the sports media experience. The potential influence of these components on the dependent variables is explored both at the overall level (average effect) and as a function of users’ characteristics (heterogeneous effects). To these aims, we conducted an experimental study exploring the influence of these immersive and interactive elements on the quality and Presence dimensions of the media experience. Along the quality dimension, we look for possible variations on the quality scores as-signed to the overall media experience and to the media components content, image, audio, sensory effects, interaction with 3D objects and interaction using the tablet device. The potential impact on Presence is analyzed by looking at two of the four dimensions defined by the ITC-SOPI questionnaire, namely Spatial Presence and Engagement. The users’ characteristics considered encompass the following personal affective, cognitive and behavioral attributes: preferences and habits in relation to the content, knowledge of the involved technologies, tendency to get emotionally involved and tendency to get absorbed in an activity and block out external distractors and the big five personality traits extraversion, agreeableness, conscientiousness, neuroticism and openness to experience. At the overall level, we found that participants preferred binaural audio than standard stereo audio and that sensory effects increase significantly the level of Spatial Presence. Several heterogeneous effects were also revealed as a result of our experimental manipulations. Interestingly, these effects were not equally distributed across the quality and Presence measures analyzed. Whereas binaural audio was foud to have an influence on the majority of the quality and Presence measures considered, the effects of sensory effects and of interaction with additional content through the tablet device concentrate mainly on the dimensions of Presence and on quality measures, respectively. The magnitude of these effects was modulated by individual’s characteristics, such as: preferences in relation to the content, frequency of viewing similar content, knowledge of involved technologies, gender, tendency to get emotionally involved, tendency to absorption and levels of agreeableness, conscientiousness and openness to experience. The personal characteristics collected in our experiment explained most of the variation in the dependent variables, confirming the frequently neglected role of individual differences on the media experience. Preferences in relation to the content, knowledge of involved technologies and tendency to get emotionally involved were among the user variables with the most generalized influence. In particular, the former two features seem to present a conflict in the allocation of attentional resources towards the media content versus the technical features of the system, respectively. Additionally, football fans’ experience seems to be modulated by emotional processes whereas for not fans, cognitive processes (and in particular those related to quality judgment) prevail.