High-Level Modeling of Multimodal Interaction Techniques Using NiMMiT

The past few years, multimodal interaction has been gaining importance in virtual environments. Although multimodality renders interacting with an environment more natural and intuitive, the development cycle of such an application is often long and expensive. In our overall field of research, we investigate how modelbased design can facilitate the development process by designing environments through the use of highlevel diagrams. In this scope, we present ‘NiMMiT’, a graphical notation for expressing and evaluating multimodal user interaction; we elaborate on the NiMMiT primitives and demonstrate its use by means of a comprehensive example.

Modelado y Evaluación de Entornos Virtuales Interactivos para Rehabilitación Cognitiva en Daño Cerebral

El Daño Cerebral (DC) se refiere a cualquier lesión producida en el cerebro y que afecta a su funcionalidad. Se ha convertido en una de las principales causas de discapacidad neurológica de las sociedades desarrolladas. Hasta la más sencilla de las actividades y acciones que realizamos en nuestro día a día involucran a los procesos cognitivos. Por ello, la alteración de las funciones cognitivas como consecuencia del DC, limita no sólo la calidad de vida del paciente sino también la de las persona de su entorno. La rehabilitación cognitiva trata de aumentar la autonomía y calidad de vida del paciente minimizando o compensando los desórdenes funciones causados por el episodio de DC. La plasticidad cerebral es una propiedad intrínseca al sistema nervioso humano por la que en función a la experiencia se crean nuevos patrones de conectividad. El propósito de la neurorrehabilitación es precisamente modular esta propiedad intrínseca a partir de ejercicios específicos, los cuales podrían derivar en la recuperación parcial o total de las funciones afectadas. La incorporación de la tecnología a las terapias de rehabilitación ha permitido desarrollar nuevas metodologías de trabajo. Esto ha ayudado a hacer frente a las dificultades de la rehabilitación que los procesos tradicionales no logran abarcar. A pesar del gran avance realizado en los Ãoltimos años, todavía existen debilidades en el proceso de rehabilitación; por ejemplo, la trasferencia a la vida real de las habilidades logradas durante la terapia de rehabilitación, así como su generalización a otras actividades cotidianas. Los entornos virtuales pueden reproducir situaciones cotidianas. Permiten simular, de forma controlada, los requisitos conductuales que encontramos en la vida real. En un contexto terapéutico, puede ser utilizado por el neuropsicólogo para corregir en el paciente comportamientos patológicos no deseados, realizar intervenciones terapéuticas sobre Actividades de Vida Diaria que estimulen conductas adaptativas. A pesar de que las tecnologías actuales tienen potencial suficiente para aportar nuevos beneficios al proceso de rehabilitación, existe cierta reticencia a su incorporación a la clínica diaria. A día de hoy, no se ha podido demostrar que su uso aporte una mejorar significativa con respecto a otro tipo de intervención; en otras palabras, no existe evidencia científica de la eficacia del uso de entornos virtuales interactivos en rehabilitación. En este contexto, la presente Tesis Doctoral trata de abordar los aspectos que mantienen a los entornos virtuales interactivos al margen de la rutina clínica diaria. Se estudian las diferentes etapas del proceso de rehabilitación cognitiva relacionado con la integración y uso de estos entornos: diseño de las actividades, su implementación en el entorno virtual, y finalmente la ejecución por el paciente y análisis de los respectivos datos. Por tanto, los bloques en los que queda dividido el trabajo de investigación expuesto en esta memoria son: 1. Diseño de las AVD. La definición y configuración de los elementos que componen la AVD permite al terapeuta diseñar estrategias de intervención terapéutica para actuar sobre el comportamiento del paciente durante la ejecución de la actividad. En esta parte de la tesis se pretende formalizar el diseño de las AVD de tal forma que el terapeuta pueda explotar el potencial tecnológico de los entornos virtuales interactivos abstrayéndose de la complejidad implícita a la tecnología. Para hacer viable este planteamiento se propone una metodología que permita modelar la definición de las AVD, representar el conocimiento implícito en ellas, y asistir al neuropsicólogo durante el proceso de diseño de la intervención clínica. 2. Entorno virtual interactivo. El gran avance tecnológico producido durante los Ãoltimos años permite reproducir AVD interactivas en un contexto de uso clínico. El objetivo perseguido en esta parte de la Tesis es el de extraer las características potenciales de esta solución tecnológica y aplicarla a las necesidades y requisitos de la rehabilitación cognitiva. Se propone el uso de la tecnología de Vídeo Interactivo para el desarrollo de estos entornos virtuales. Para la evaluación de la misma se realiza un estudio experimental dividido en dos fases con la participación de sujetos sanos y pacientes, donde se valora su idoneidad para ser utilizado en terapias de rehabilitación cognitiva. 3. Monitorización de las AVD. El uso de estos entornos virtuales interactivos expone al paciente ante una gran cantidad de estímulos e interacciones. Este hecho requiere de instrumentos de monitorización avanzado que aporten al terapeuta información objetiva sobre el comportamiento del paciente, lo que le podría permitir por ejemplo evaluar la eficacia del tratamiento. En este apartado se propone el uso de métricas basadas en la atención visual y la interacción con el entorno para conocer datos sobre el comportamiento del paciente durante la AVD. Se desarrolla un sistema de monitorización integrado con el entorno virtual que ofrece los instrumentos necesarios para la evaluación de estas métricas para su uso clínico. La metodología propuesta ha permitido diseñar una AVD basada en la definición de intervenciones terapéuticas. Posteriormente esta AVD has sido implementada mediante la tecnología de vídeo interactivo, creando así el prototipo de un entorno virtual para ser utilizado por pacientes con déficit cognitivo. Los resultados del estudio experimental mediante el cual ha sido evaluado demuestran la robustez y usabilidad del sistema, así como su capacidad para intervenir sobre el comportamiento del paciente. El sistema monitorización que ha sido integrado con el entorno virtual aporta datos objetivos sobre el comportamiento del paciente durante la ejecución de la actividad. Los resultados obtenidos permiten contrastar las hipótesis de investigación planteadas en la Tesis Doctoral, aportando soluciones que pueden ayudar a la integración de los entornos virtuales interactivos en la rutina clínica. Esto abre una nueva vía de investigación y desarrollo que podría suponer un gran progreso y mejora en los procesos de neurorrehabilitación cognitiva en daño cerebral. ABSTRACT Brain injury (BI) refers to medical conditions that occur in the brain, altering its function. It becomes one of the main neurological disabilities in the developed society. Cognitive processes determine individual performance in Activities of Daily Living (ADL), thus, the cognitive disorders after BI result in a loss of autonomy and independence, affecting the patient’s quality of life. Cognitive rehabilitation seeks to increase patients’ autonomy and quality of life minimizing or compensating functional disorders showed by BI patients. Brain plasticity is an intrinsic property of the human nervous system whereby its structure is changed depending on experience. Neurorehabilitation pursuits a precise modulation of this intrinsic property, based on specific exercises to induce functional changes, which could result in partial or total recovery of the affected functions. The new methodologies that can be approached by applying technologies to the rehabilitation process, permit to deal with the difficulties which are out of the scope of the traditional rehabilitation. Despite this huge breakthrough, there are still weaknesses in the rehabilitation process, such as the transferring to the real life those skills reached along the therapy, and its generalization to others daily activities. Virtual environments reproduce daily situations. Behavioural requirements which are similar to those we perceive in real life, are simulated in a controlled way. In these virtual environments the therapist is allowed to interact with patients without even being present, inhibiting unsuitable behaviour patterns, stimulating correct answers throughout the simulation and enhancing stimuli with supplementary information when necessary. Despite the benefits which could be brought to the cognitive rehabilitation by applying the potential of the current technologies, there are barriers for widespread use of interactive virtual environments in clinical routine. At present, the evidence that these technologies bring a significant improvement to the cognitive therapies is limited. In other words, there is no evidence about the efficacy of using virtual environments in rehabilitation. In this context, this work aims to address those issues which keep the virtual environments out of the clinical routine. The stages of the cognitive rehabilitation process, which are related with the use and integration of these environments, are analysed: activities design, its implementation in the virtual environment, and the patient’s performance and the data analysis. Hence, the thesis is comprised of the main chapters that are listed below: 1. ADL Design.Definition and configuration of the elements which comprise the ADL allow the therapist to design intervention strategies to influence over the patient behaviour along the activity performance. This chapter aims to formalise the AVD design in order to help neuropsychologists to make use of the interactive virtual environments’ potential but isolating them from the complexity of the technology. With this purpose a new methodology is proposed as an instrument to model the ADL definition, to manage its implied knowledge and to assist the clinician along the design process of the therapeutic intervention. 2. Interactive virtual environment. Continuous advancements make the technology feasible for re-creating rehabilitation therapies based on ADL. The goal of this stage is to analyse the main features of virtual environments in order to apply them according to the cognitive rehabilitation’s requirements. The interactive video is proposed as the technology to develop virtual environments. Experimental study is carried out to assess the suitability of the interactive video to be used by cognitive rehabilitation. 3. ADL monitoring system. This kind of virtual environments bring patients in front lots of stimuli and interactions. Thus, advanced monitoring instruments are needed to provide therapist with objective information about patient’s behaviour. This thesis chapter propose the use of metrics rely on visual patients’ visual attention and their interactions with the environment. A monitoring system has been developed and integrated with the interactive video-based virtual environment, providing neuropsychologist with the instruments to evaluate the clinical force of this metrics. Therapeutic interventions-based ADL has been designed by using the proposed methodology. Interactive video technology has been used to develop the ADL, resulting in a virtual environment prototype to be use by patients who suffer a cognitive deficits. An experimental study has been performed to evaluate the virtual environment, whose overcomes show the usability and solidity of the system, and also its capacity to have influence over patient’s behaviour. The monitoring system, which has been embedded in the virtual environment, provides objective information about patients’ behaviour along their activity performance. Research hypothesis of the Thesis are proven by the obtained results. They could help to incorporate the interactive virtual environments in the clinical routine. This may be a significant step forward to enhance the cognitive neurorehabilitation processes in brain injury.

Students' collective knowledge construction in the virtual learning environment ""ToLigado - your school interactive newspaper""

Introduction. The ToLigado Project - Your School Interactive Newspaper is an interactive virtual learning environment conceived, developed, implemented and supported by researchers at the School of the Future Research Laboratory of the University of Sao Paulo, Brazil. Method. This virtual learning environment aims to motivate trans-disciplinary research among public school students and teachers in 2,931 schools equipped with Internet-access computer rooms. Within this virtual community, students produce collective multimedia research documents that are immediately published in the portal. The project also aims to increase students' autonomy for research, collaborative work and Web authorship. Main sections of the portal are presented and described. Results. Partial results of the first two years' implementation are presented and indicate a strong motivation among students to produce knowledge despite the fragile hardware and software infrastructure at the time. Discussion. In this new environment, students should be seen as 'knowledge architects' and teachers as facilitators, or 'curiosity managers'. The ToLigado portal may constitute a repository for future studies regarding student attitudes in virtual learning environments, students' behaviour as 'authors', Web authorship involving collective knowledge production, teachers' behaviour as facilitators, and virtual learning environments as digital repositories of students' knowledge construction and social capital in virtual learning communities.

The Virtual Interactive Relationship Between BIM Project Teams: Effective Communication to aid Collaboration in the Design Process

Building Information Modelling (BIM) provides a shared source of information about a built asset, which creates a collaborative virtual environment for project teams. Literature suggests that to collaborate efficiently, the relationship between the project team is based on sympathy, obligation, trust and rapport. Communication increases in importance when working collaboratively but effective communication can only be achieved when the stakeholders are willing to act, react, listen and share information. Case study research and interviews with Architecture, Engineering and Construction (AEC) industry experts suggest that synchronous face-to-face communication is project teams’ preferred method, allowing teams to socialise and build rapport, accelerating the creation of trust between the stakeholders. However, virtual unified communication platforms are a close second-preferred option for communication between the teams. Effective methods for virtual communication in professional practice, such as virtual collaboration environments (CVE), that build trust and achieve similar spontaneous responses as face-to-face communication, are necessary to face the global challenges and can be achieved with the right people, processes and technology. This research paper investigates current industry methods for virtual communication within BIM projects and explores the suitability of avatar interaction in a collaborative virtual environment as an alternative to face-to-face communication to enhance collaboration between design teams’ professional practice on a project. Hence, this paper presents comparisons between the effectiveness of these communication methods within construction design teams with results of further experiments conducted to test recommendations for more efficient methods for virtual communication to add value in the workplace between design teams.

3D Environment for Virtual Collections

This paper describes a framework for building virtual collections of several digital objects and presenting them in an interactive 3D environment, rendered in a web browser. Using that environment, the website visitor can examine a given collection from a first-person perspective by walking around and inspecting each object in detail by viewing it from any angle. The rendering and visualization of the models is done solely by the web browser with the use of HTML5 and the Three.js JavaScript library, without any additional requirements.

A survey of techniques and technologies for web-based real-time interactive rendering

When exploring a virtual environment, realism depends mainly on two factors: realistic images and real-time feedback (motions, behaviour etc.). In this context, photo realism and physical validity of computer generated images required by emerging applications, such as advanced e-commerce, still impose major challenges in the area of rendering research whereas the complexity of lighting phenomena further requires powerful and predictable computing if time constraints must be attained. In this technical report we address the state-of-the-art on rendering, trying to put the focus on approaches, techniques and technologies that might enable real-time interactive web-based clientserver rendering systems. The focus is on the end-systems and not the networking technologies used to interconnect client(s) and server(s).

Automatic camera control in virtual environments augmented using multiple sparse videos

Automated virtual camera control has been widely used in animation and interactive virtual environments. We have developed a multiple sparse camera based free view video system prototype that allows users to control the position and orientation of a virtual camera, enabling the observation of a real scene in three dimensions (3D) from any desired viewpoint. Automatic camera control can be activated to follow selected objects by the user. Our method combines a simple geometric model of the scene composed of planes (virtual environment), augmented with visual information from the cameras and pre-computed tracking information of moving targets to generate novel perspective corrected 3D views of the virtual camera and moving objects. To achieve real-time rendering performance, view-dependent textured mapped billboards are used to render the moving objects at their correct locations and foreground masks are used to remove the moving objects from the projected video streams. The current prototype runs on a PC with a common graphics card and can generate virtual 2D views from three cameras of resolution 768 x 576 with several moving objects at about 11 fps. (C)2011 Elsevier Ltd. All rights reserved.

Participatory design in the project of virtual learning environment of histology

This present article describes a research on the development, under the approach of participatory design, a virtual teaching-learning of Histology in which students and teachers participated actively in all stages of development of the educational environment. We postulates that the development of virtual learning environment of Histology, through the Participatory Design approach, contributes to greater acceptance and use by students and that the adoption of virtual environment for teaching and learning by teachers is a determining factor of use by students

Mediação do processo ensino-aprendizagem com o jogo de empresas mercado virtual: uma pesquisa de opinião

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Elaboração e avaliação de um ambiente virtual para o ensino/aprendizagem de embriologia

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

An interactive surgical planning tool for acetabular fractures: initial results

Background Acetabular fractures still are among the most challenging fractures to treat because of complex anatomy, involved surgical access to fracture sites and the relatively low incidence of these lesions. Proper evaluation and surgical planning is necessary to achieve anatomic reduction of the articular surface and stable fixation of the pelvic ring. The goal of this study was to test the feasibility of preoperative surgical planning in acetabular fractures using a new prototype planning tool based on an interactive virtual reality-style environment. Methods 7 patients (5 male and 2 female; median age 53 y (25 to 92 y)) with an acetabular fracture were prospectively included. Exclusion criterions were simple wall fractures, cases with anticipated surgical dislocation of the femoral head for joint debridement and accurate fracture reduction. According to the Letournel classification 4 cases had two column fractures, 2 cases had anterior column fractures and 1 case had a T-shaped fracture including a posterior wall fracture. The workflow included following steps: (1) Formation of a patient-specific bone model from preoperative computed tomography scans, (2) interactive virtual fracture reduction with visuo-haptic feedback, (3) virtual fracture fixation using common osteosynthesis implants and (4) measurement of implant position relative to landmarks. The surgeon manually contoured osteosynthesis plates preoperatively according to the virtually defined deformation. Screenshots including all measurements for the OR were available. The tool was validated comparing the preoperative planning and postoperative results by 3D-superimposition. Results Preoperative planning was feasible in all cases. In 6 of 7 cases superimposition of preoperative planning and postoperative follow-up CT showed a good to excellent correlation. In one case part of the procedure had to be changed due to impossibility of fracture reduction from an ilioinguinal approach. In 3 cases with osteopenic bone patient-specific prebent fixation plates were helpful in guiding fracture reduction. Additionally, anatomical landmark based measurements were helpful for intraoperative navigation. Conclusion The presented prototype planning tool for pelvic surgery was successfully integrated in a clinical workflow to improve patient-specific preoperative planning, giving visual and haptic information about the injury and allowing a patient-specific adaptation of osteosynthesis implants to the virtually reduced pelvis.

Presence in a Three-Dimensional Test Environment: Benefit or Threat to Market Research?

In market research, the adoption of interactive virtual reality-techniques could be expected to contain many advantages: artificial lab environments could be designed in a more realistic manner and the consideration of “time to the market”-factors could be improved. On the other hand, with an increasing degree of presence and the notional attendance in a simulated test environment, the market research task could fall prey to the tensing virtual reality adventure. In the following study a 3D-technique is empirically tested for its usability in market research. It will be shown that the interactive 3D-simulation is not biased by the immersion it generates and provides considerably better test results than 2D-stimuli do.

Language Learning in 3D Virtual World

Second Life (SL) is an ideal platform for language learning. It is called a Multi-User Virtual Environment, where users can have varieties of learning experiences in life-like environments. Numerous attempts have been made to use SL as a platform for language teaching and the possibility of SL as a means to promote conversational interactions has been reported. However, the research so far has largely focused on simply using SL without further augmentations for communication between learners or between teachers and learners in a school-like environment. Conversely, not enough attention has been paid to its controllability which builds on the embedded functions in SL. This study, based on the latest theories of second language acquisition, especially on the Task Based Language Teaching and the Interaction Hypothesis, proposes to design and implement an automatized interactive task space (AITS) where robotic agents work as interlocutors of learners. This paper presents a design that incorporates the SLA theories into SL and the implementation method of the design to construct AITS, fulfilling the controllability of SL. It also presents the result of the evaluation experiment conducted on the constructed AITS.

A graphical tool for designing interactive video cognitive rehabilitation therapies

Acquired Brain Injury (ABI) has become one of the most common causes of neurological disability in developed countries. Cognitive disorders result in a loss of independence and therefore patients? quality of life. Cognitive rehabilitation aims to promote patients? skills to achieve their highest degree of personal autonomy. New technologies such as interactive video, whereby real situations of daily living are reproduced within a controlled virtual environment, enable the design of personalized therapies with a high level of generalization and a great ecological validity. This paper presents a graphical tool that allows neuropsychologists to design, modify, and configure interactive video therapeutic activities, through the combination of graphic and natural language. The tool has been validated creating several Activities of Daily Living and a preliminary usability evaluation has been performed showing a good clinical acceptance in the definition of complex interactive video therapies for cognitive rehabilitation.

Meaningful Interaction in Virtual Learning Environments

There is still a great deal of opportunity for research on contextual interactive immersion in virtual heritage environments. The general failure of virtual environment technology to create engaging and educational experiences may be attributable not just to deficiencies in technology or in visual fidelity, but also to a lack of contextual and performative-based interaction, such as that found in games. However, there is little written so far on exactly how game-style interaction can help improve virtual learning environments.