Impact Study of Nonverbal Facial Cues on Spontaneous Chatting with Virtual Humans

Non-verbal communication (NVC) is considered to represent more than 90 percent of everyday communication. In virtual world, this important aspect of interaction between virtual humans (VH) is strongly neglected. This paper presents a user-test study to demonstrate the impact of automatically generated graphics-based NVC expression on the dialog quality: first, we wanted to compare impassive and emotion facial expression simulation for impact on the chatting. Second, we wanted to see whether people like chatting within a 3D graphical environment. Our model only proposes facial expressions and head movements induced from spontaneous chatting between VHs. Only subtle facial expressions are being used as nonverbal cues - i.e. related to the emotional model. Motion capture animations related to hand gestures, such as cleaning glasses, were randomly used to make the virtual human lively. After briefly introducing the technical architecture of the 3D-chatting system, we focus on two aspects of chatting through VHs. First, what is the influence of facial expressions that are induced from text dialog? For this purpose, we exploited an emotion engine extracting an emotional content from a text and depicting it into a virtual character developed previously [GAS11]. Second, as our goal was not addressing automatic generation of text, we compared the impact of nonverbal cues in conversation with a chatbot or with a human operator with a wizard of oz approach. Among main results, the within group study -involving 40 subjects- suggests that subtle facial expressions impact significantly not only on the quality of experience but also on dialog understanding.

Application of Time-Delay Estimation to Mixed Reality Multisensor Tracking

Spatial tracking is one of the most challenging and important parts of Mixed Reality environments. Many applications, especially in the domain of Augmented Reality, rely on the fusion of several tracking systems in order to optimize the overall performance. While the topic of spatial tracking sensor fusion has already seen considerable interest, most results only deal with the integration of carefully arranged setups as opposed to dynamic sensor fusion setups. A crucial prerequisite for correct sensor fusion is the temporal alignment of the tracking data from several sensors. Tracking sensors are typically encountered in Mixed Reality applications, are generally not synchronized. We present a general method to calibrate the temporal offset between different sensors by the Time Delay Estimation method which can be used to perform on-line temporal calibration. By applying Time Delay Estimation on the tracking data, we show that the temporal offset between generic Mixed Reality spatial tracking sensors can be calibrated. To show the correctness and the feasibility of this approach, we have examined different variations of our method and evaluated various combinations of tracking sensors. We furthermore integrated this time synchronization method into our UBITRACK Mixed Reality tracking framework to provide facilities for calibration and real-time data alignment.

Learning in Virtual Worlds: Research and Applications

Three-dimensional (3D) immersive virtual worlds have been touted as being capable of facilitating highly interactive, engaging, multimodal learning experiences. Much of the evidence gathered to support these claims has been anecdotal but the potential that these environments hold to solve traditional problems in online and technology-mediated education—primarily learner isolation and student disengagement—has resulted in considerable investments in virtual world platforms like Second Life, OpenSimulator, and Open Wonderland by both professors and institutions. To justify this ongoing and sustained investment, institutions and proponents of simulated learning environments must assemble a robust body of evidence that illustrates the most effective use of this powerful learning tool. In this authoritative collection, a team of international experts outline the emerging trends and developments in the use of 3D virtual worlds for teaching and learning. They explore aspects of learner interaction with virtual worlds, such as user wayfinding in Second Life, communication modes and perceived presence, and accessibility issues for elderly or disabled learners. They also examine advanced technologies that hold potential for the enhancement of learner immersion and discuss best practices in the design and implementation of virtual world-based learning interventions and tasks. By evaluating and documenting different methods, approaches, and strategies, the contributors to Learning in Virtual Worlds offer important information and insight to both scholars and practitioners in the field. AU Press is an open access publisher and the book is available for free in PDF format as well as for purchase on our website: http://bit.ly/1W4yTRA

Tecnología de registro y reproducción sonora para entornos de realidad virtual

Este proyecto de fin de carrera tiene como objetivo obtener una visión detallada de los sistemas y tecnologías de grabación y reproducción utilizadas para aplicaciones de audio 3D y entornos de realidad virtual, analizando las diferentes alternativas existentes, su funcionamiento, características, detalles técnicos y sus ámbitos de aplicación. Como punto de partida se estudiará la teoría psicoacústica y la localización de fuentes sonoras en el espacio, base para el estudio de los sistemas de audio 3D. Se estudiará tanto la espacialización sonora en un espacio real y la espacialización virtual (simulación mediante procesado de información de la localización de fuentes sonoras), en los que intervienen algunos fenómenos acústicos y psicoacústicos como ITD, o diferencia de tiempo que existe entre una señal acústica que llega a los pabellones auditivos, la ILD, o diferencia de intensidad o amplitud que hay entre la señal que llega a los pabellones auditivos y la localización espacial mediante otra serie de mecanismos biaurales. Tras una visión general de la teoría psicoacústica y la espacialización sonora, se analizarán con detalle los elementos de grabación y reproducción existentes para audio 3D. Concretamente, a lo largo del proyecto se profundizará en el funcionamiento del sistema estéreo, caracterizado por el posicionamiento sonoro mediante la utilización de dos canales; del sistema biaural, caracterizado por reconstruir campos sonoros mediante el uso de las HRTF; de los sistemas multicanal, detallando gran parte de las alternativas y configuraciones existentes; del sistema Ambiophonics, caracterizado por implementar filtros de cruce; del sistema Ambisonics, y sus diferentes formatos y técnicas de codificación y decodificación; y del sistema Wavefield Synthesis, caracterizado por recrear ambientes sonoros en grandes espacios. ABSTRACT This project aims to get a detailed view of recording and reproducing systems and technologies used to 3D audio applications and virtual reality environments, analyzing the different alternatives available, their functioning, features, technical details and their different scopes of applications. As a starting point, will be studied the psychoacoustic theory and the localization of sound sources in space, basis for the 3D audio study. Will be studied both the spacialization of sound sources in real space as virtual spatialization of sound sources (simulation by information processing of localization of sound sources), in which involves some acoustic and psychoacoustic phenomena like ITD (or the Interaural time difference), the ILD, (or the Interaural Level Difference) and spatial localization by another set of binaural mechanisms. After a general overview of the psychoacoustics theory and the sound spatialization, will be analyzed in detail existing methods of recording and reproducing for 3D audio. Specifically, during the project will analyze the characteristics of the stereo systems, characterized by sound positioning using two channels; the binaural systems, characterized by reconstructing sound fields by using the HRTF; the multichannel systems, detailing many of the existing alternatives and configurations; the Ambiophonics system, which is characterized by implementing crosstalk elimination techniques; the Ambiosonics system, and its various formats and encoding and decoding techniques; and the Wavefield Synthesis system, characterized by recreate soundscapes in large spaces.

Evaluación y tratamiento basados en realidad virtual de la negligencia espacial unilateral: análisis del estado de la cuestión y demostración de enfoques alternativos

En el siguiente trabajo se presenta en primer lugar de forma detallada la enfermedad denominada negligencia espacial unilateral (síntomas, tipos, causas, evaluación y tratamientos) para proporcionar una mejor comprensión del principal objetivo del estudio, que es el análisis de las soluciones virtuales, existentes en la literatura, aplicadas al tratamiento de esta enfermedad, incluyéndose una amplia descripción de cada estudio encontrado sobre el tema. A continuación, se han realizado tres implementaciones en realidad virtual de tres técnicas clásicas de rehabilitación llevadas a cabo en un entorno virtual, que son la estimulación optocinética, eye patching, y adaptación prismática y se ha desarrollado una aplicación 3D para evaluar el grado y tipo de negligencia sufrida por los pacientes. Que de forma conjunta, constituyen un primer paso hacia un enfoque alternativo para el tratamiento de la enfermedad, más personalizado y eficaz. Por último, en las conclusiones, se analizan las principales ventajas y desventajas encontradas en el uso de estas tecnologías aplicadas a la enfermedad y los trabajos futuros que pueden derivar de este trabajo.---ABSTRACT---The following work starts by presenting in detail a disease called unilateral spatial neglect (symptoms, types, causes, assessment and treatment) to provide the background for this study's main objective, which is the analysis of the virtual solutions existing in the literature for the treatment of this disease. The document includes an extensive description of the previous work found in this topic. Afterwards, three implementations of three classical rehabilitation techniques were performed in virtual reality: optokinetic stimulation, eye patching and prism adaptation, as a proof-of-concept, and a 3D application was implemented to assess the degree and type of negligence suffered by patients. Altogether, they constitute a first step towards an alternative approach for the treatment of disease, more personalized and effective. Finally, the conclusions analyze the main advantages and disadvantages encountered in the use of these technologies when applied to this disease and suggest future work.

Restauración virtual del patrimonio sonoro. Aplicación al antiguo rito hispánico

Una de las aplicaciones más interesantes de las tecnologías de Realidad Acústica Virtual es la posibilidad de recuperar el patrimonio cultural del sonido de los recintos históricos que se han perdido o cuyas características se han modificado a lo largo de los siglos. En este trabajo, las tecnologías de realidad acústica virtual se utilizan para tratar de reconstruir virtualmente el sonido producido en las actividades litúrgicas del antiguo rito hispánico. Bajo este nombre se conoce a la liturgia que celebraban los cristianos de la península ibérica hasta su prohibición, en favor del culto romano, a mediados del siglo XI. El rito hispánico -también conocido como rito visigótico-mozárabe- es una de las manifestaciones culturales más interesantes de la Alta Edad Media en occidente. Se trata de un patrimonio cultural desaparecido, tanto desde el punto de vista de las señales sonoras que lo integraban -ya que la mayoría de las melodías que conformaban el rito se han perdido- como desde el punto de vista de los espacios en los que se desarrollaba, debido a que las iglesias conservadas de la época han experimentado modificaciones a lo largo de los siglos que alteran sus condiciones acústicas con respecto a las que tenían en el periodo de vigencia de esta liturgia. Para llevar a cabo este proyecto, se han realizado modelos acústicos digitales de un grupo representativo de iglesias prerrománicas de la Península Ibérica en su estado primitivo. Se ha procurado que las iglesias seleccionadas representen la variedad de comportamientos acústicos esperables en este tipo de edificios. Con este objetivo, se han elegido cinco iglesias prerrománicas que presentan diferencias sustanciales en los parámetros que, a priori, van a influir en mayor manera a su comportamiento acústico: el volumen del recinto, la forma de la planta y el tipo de cubierta. El proceso de creación de los modelos acústicos digitales de las iglesias se ha dividido en dos fases: en la primera se han creado modelos de los edificios en su estado actual, que se han validado a partir de los datos obtenidos en mediciones acústicas realizadas in situ; a partir de los modelos validados de las iglesias en su estado actual, en la segunda fase se han generado los modelos acústicos digitales correspondientes al estado primitivo de las mismas, modificando los modelos anteriores de acuerdo con las hipótesis de reconstrucción propuestas en las investigaciones arqueológicas más recientes. Se han realizado grabaciones en cámara anecoica de una serie de piezas del repertorio original del canto mozárabe. Las grabaciones se han llevado a cabo con un array esférico compuesto por 32 micrófonos, con objeto de obtener información sobre la directividad de la emisión sonora de los cantantes, que se aplicará en las auralizaciones. Finalmente, se han realizado diversas auralizaciones, teniendo en cuenta diferentes configuraciones litúrgicas que eran usuales en este rito. ABSTRACT One of the most interesting applications of the Acoustic Virtual Reality technologies is the possibility to recover the cultural heritage of the sound of the historical sites that have been lost or whose characteristics have been modified through time. In this work, Acoustic Virtual Reality technologies are used to try to reconstruct virtually the sound produced in the liturgical activities of the Hispanic Rite. This is the name given to the liturgy celebrated by Christians of the Iberian Peninsula prior to the introduction of the Roman cult in the mid-eleventh century. The Hispanic Rite (also known as Visigothic or Mozarabic rite) is one of the most interesting cultural manifestations of the Middle Ages. It is a lost cultural heritage, both from the point of view of the sound signals that were used since the majority of the melodies that conformed the rite have been lost, and from the point of view of the spaces in which this liturgy was celebrated, because the churches preserved from that era have changed over the centuries altering its acoustic conditions respect to which they were in the period in which this liturgy was in effect. To carry out this project, acoustic models of a representative group of the pre-Romanesque churches in the Iberian Peninsula have been made in his primitive state. To select the sample of buildings to study, efforts have been made to ensure that the selected churches were representative of the range of expected acoustic behaviors in this type of buildings. Five churches have been selected, showing differences in the parameters that have the greatest influence on their acoustic behavior: the enclosure volume, the shape of the floor plan and the type of roof. The process of creating digital acoustic models of the churches has been divided into two phases. In the first phase acoustic models of the churches in its current state have been created. These models have been validated with the data obtained from in situ acoustic measurements. From the validated models of churches in its current state, in the second phase, changes in the acoustic models have been performed in order to represent the primitive state of the churches, according to the historical reconstruction hypothesis that have been proposed in the most recent archaeological investigations. Anechoic recordings of a series of pieces of the original Mozarabic Chant repertoire have been recorded. The recordings were made with a spherical array composed of 32 microphones, in order to obtain information on the directivity of the sound emission of the singers. These directivity data will be used to simulate the directional radiation of the sound sources in the auralizations. Finally, auralizations were produced corresponding to different liturgical configurations which were common in this rite.

Modelling ecological cognitive rehabilitation therapies for building virtual environments in brain injury

Brain Injury (BI) has become one of the most common causes of neurological disability in developed countries. Cognitive disorders result in a loss of independence and patients? quality of life. Cognitive rehabilitation aims to promote patients? skills to achieve their highest degree of personal autonomy. New technologies such as virtual reality or interactive video allow developing rehabilitation therapies based on reproducible Activities of Daily Living (ADLs), increasing the ecological validity of the therapy. However, the lack of frameworks to formalize and represent the definition of this kind of therapies can be a barrier for widespread use of interactive virtual environments in clinical routine.

Inclusion of immersive virtual learning environments and visual control systems to support the learning of students with Asperger syndrome

This paper presents the use of immersive virtual reality systems in the educational intervention with Asperger students. The starting points of this study are features of these students' cognitive style that requires an explicit teaching style supported by visual aids and highly structured environments. The proposed immersive virtual reality system, not only to assess the student's behavior and progress, but also is able to adapt itself to the student's specific needs. Additionally, the immersive reality system is equipped with sensors that can determine certain behaviors of the students. This paper determines the possible inclusion of immersive virtual reality as a support tool and learning strategy in these particular students' intervention. With this objective two task protocols have been defined with which the behavior and interaction situations performed by participant students are recorded. The conclusions from this study talks in favor of the inclusion of these virtual immersive environments as a support tool in the educational intervention of Asperger syndrome students as their social competences and executive functions have improved.

Virtual training environments to improve train driver's crisis decision making

This paper reports on a current research project in which virtual reality simulators are being investigated as a means of simulating hazardous Rail work conditions in order to allow train drivers to practice decision-making under stress. When working under high stress conditions train drivers need to move beyond procedural responses into a response activated through their own problem-solving and decision-making skills. This study focuses on the use of stress inoculation training which aims to build driver’s confidence in the use of new decision-making skills by being repeatedly required to respond to hazardous driving conditions. In particular, the study makes use of a train cab driving simulator to reproduce potentially stress inducing real-world scenarios. Initial pilot research has been undertaken in which drivers have experienced the training simulation and subsequently completed surveys on the level of immersion experienced. Concurrently drivers have also participated in a velocity perception experiment designed to objectively measure the fidelity of the virtual training environment. Baseline data, against which decision-making skills post training will be measured, is being gathered via cognitive task analysis designed to identify primary decision requirements for specific rail events. While considerable efforts have been invested in improving Virtual Reality technology, little is known about how to best use this technology for training personnel to respond to workplace conditions in the Rail Industry. To enable the best use of simulators for training in the Rail context the project aims to identify those factors within virtual reality that support required learning outcomes and use this information to design training simulations that reliably and safely train staff in required workplace accident response skills.

Enhancing experiential learning via virtual environments

Research indicates Virtual Reality (VR) is delivering on it's promised potential to provide enhanced training and education outcomes. A significant research project, at the University of Queensland, has constructed a number of virtual contexts in which the phenomena experienced by patients who have psychosis are reproduced for use in psychiatry education. Symptoms of psychosis reproduced include delusions, hallucinations and thought disorder. The new software enables psychiatry students to experience the inner world of a patient with psychosis. Lecturers in psychiatry report VR has the potential to enhance student's abilities to actually 'feel' the types of emotions and physiological reactions a hallucination precipitates in a patient. The current work of the project and stages of software development will be demonstrated. The virtual environments provide a new method of delivering experiential learning opportunities to higher education classrooms.