The MIRELA framework: modeling and analyzing mixed reality applications using timed automata

Mixed Reality (MR) aims to link virtual entities with the real world and has many applications such as military and medical domains [JBL+00, NFB07]. In many MR systems and more precisely in augmented scenes, one needs the application to render the virtual part accurately at the right time. To achieve this, such systems acquire data related to the real world from a set of sensors before rendering virtual entities. A suitable system architecture should minimize the delays to keep the overall system delay (also called end-to-end latency) within the requirements for real-time performance. In this context, we propose a compositional modeling framework for MR software architectures in order to specify, simulate and validate formally the time constraints of such systems. Our approach is first based on a functional decomposition of such systems into generic components. The obtained elements as well as their typical interactions give rise to generic representations in terms of timed automata. A whole system is then obtained as a composition of such defined components. To write specifications, a textual language named MIRELA (MIxed REality LAnguage) is proposed along with the corresponding compilation tools. The generated output contains timed automata in UPPAAL format for simulation and verification of time constraints. These automata may also be used to generate source code skeletons for an implementation on a MR platform. The approach is illustrated first on a small example. A realistic case study is also developed. It is modeled by several timed automata synchronizing through channels and including a large number of time constraints. Both systems have been simulated in UPPAAL and checked against the required behavioral properties.

Head Tracking Based Avatar Control for Virtual Environment Teamwork Training

Virtual environments (VE) are gaining in popularity and are increasingly used for teamwork training purposes, e.g., for medical teams. One shortcoming of modern VEs is that nonverbal communication channels, essential for teamwork, are not supported well. We address this issue by using an inexpensive webcam to track the user's head. This tracking information is used to control the head movement of the user's avatar, thereby conveying head gestures and adding a nonverbal communication channel. We conducted a user study investigating the influence of head tracking based avatar control on the perceived realism of the VE and on the performance of a surgical teamwork training scenario. Our results show that head tracking positively influences the perceived realism of the VE and the communication, but has no major influence on the training outcome.

Investigations into Velocity and Distance Perception Based on Different Types of Moving Sound Sources with Respect to Auditory Virtual Environments

The characteristics of moving sound sources have strong implications on the listener's distance perception and the estimation of velocity. Modifications of the typical sound emissions as they are currently occurring due to the tendency towards electromobility have an impact on the pedestrian's safety in road traffic. Thus, investigations of the relevant cues for velocity and distance perception of moving sound sources are not only of interest for the psychoacoustic community, but also for several applications, like e.g. virtual reality, noise pollution and safety aspects of road traffic. This article describes a series of psychoacoustic experiments in this field. Dichotic and diotic stimuli of a set of real-life recordings taken from a passing passenger car and a motorcycle were presented to test subjects who in turn were asked to determine the velocity of the object and its minimal distance from the listener. The results of these psychoacoustic experiments show that the estimated velocity is strongly linked to the object's distance. Furthermore, it could be shown that binaural cues contribute significantly to the perception of velocity. In a further experiment, it was shown that - independently of the type of the vehicle - the main parameter for distance determination is the maximum sound pressure level at the listener's position. The article suggests a system architecture for the adequate consideration of moving sound sources in virtual auditory environments. Virtual environments can thus be used to investigate the influence of new vehicle powertrain concepts and the related sound emissions of these vehicles on the pedestrians' ability to estimate the distance and velocity of moving objects.

Generating Realistic Camera Shake for Virtual Scenes

When depicting both virtual and physical worlds, the viewer's impression of presence in these worlds is strongly linked to camera motion. Plausible and artist-controlled camera movement can substantially increase scene immersion. While physical camera motion exhibits subtle details of position, rotation, and acceleration, these details are often missing for virtual camera motion. In this work, we analyze camera movement using signal theory. Our system allows us to stylize a smooth user-defined virtual base camera motion by enriching it with plausible details. A key component of our system is a database of videos filmed by physical cameras. These videos are analyzed with a camera-motion estimation algorithm (structure-from-motion) and labeled manually with a specific style. By considering spectral properties of location, orientation and acceleration, our solution learns camera motion details. Consequently, an arbitrary virtual base motion, defined in any conventional animation package, can be automatically modified according to a user-selected style. In an animation package the camera motion base path is typically defined by the user via function curves. Another possibility is to obtain the camera path by using a mixed reality camera in motion capturing studio. As shown in our experiments, the resulting shots are still fully artist-controlled, but appear richer and more physically plausible.

Impact Study of Nonverbal Facial Cues on Spontaneous Chatting with Virtual Humans

Non-verbal communication (NVC) is considered to represent more than 90 percent of everyday communication. In virtual world, this important aspect of interaction between virtual humans (VH) is strongly neglected. This paper presents a user-test study to demonstrate the impact of automatically generated graphics-based NVC expression on the dialog quality: first, we wanted to compare impassive and emotion facial expression simulation for impact on the chatting. Second, we wanted to see whether people like chatting within a 3D graphical environment. Our model only proposes facial expressions and head movements induced from spontaneous chatting between VHs. Only subtle facial expressions are being used as nonverbal cues - i.e. related to the emotional model. Motion capture animations related to hand gestures, such as cleaning glasses, were randomly used to make the virtual human lively. After briefly introducing the technical architecture of the 3D-chatting system, we focus on two aspects of chatting through VHs. First, what is the influence of facial expressions that are induced from text dialog? For this purpose, we exploited an emotion engine extracting an emotional content from a text and depicting it into a virtual character developed previously [GAS11]. Second, as our goal was not addressing automatic generation of text, we compared the impact of nonverbal cues in conversation with a chatbot or with a human operator with a wizard of oz approach. Among main results, the within group study -involving 40 subjects- suggests that subtle facial expressions impact significantly not only on the quality of experience but also on dialog understanding.

Hands-Free Navigation in Immersive Environments for the Evaluation of the Effectiveness of Indoor Navigation Systems

While navigation systems for cars are in widespread use, only recently, indoor navigation systems based on smartphone apps became technically feasible. Hence tools in order to plan and evaluate particular designs of information provision are needed. Since tests in real infrastructures are costly and environmental conditions cannot be held constant, one must resort to virtual infrastructures. This paper presents the development of an environment for the support of the design of indoor navigation systems whose center piece consists in a hands-free navigation method using the Microsoft Kinect in the four-sided Definitely Affordable Virtual Environment (DAVE). Navigation controls using the user's gestures and postures as the input to the controls are designed and implemented. The installation of expensive and bulky hardware like treadmills is avoided while still giving the user a good impression of the distance she has traveled in virtual space. An advantage in comparison to approaches using a head mounted display is that the DAVE allows the users to interact with their smartphone. Thus the effects of different indoor navigation systems can be evaluated already in the planning phase using the resulting system

Application of Time-Delay Estimation to Mixed Reality Multisensor Tracking

Spatial tracking is one of the most challenging and important parts of Mixed Reality environments. Many applications, especially in the domain of Augmented Reality, rely on the fusion of several tracking systems in order to optimize the overall performance. While the topic of spatial tracking sensor fusion has already seen considerable interest, most results only deal with the integration of carefully arranged setups as opposed to dynamic sensor fusion setups. A crucial prerequisite for correct sensor fusion is the temporal alignment of the tracking data from several sensors. Tracking sensors are typically encountered in Mixed Reality applications, are generally not synchronized. We present a general method to calibrate the temporal offset between different sensors by the Time Delay Estimation method which can be used to perform on-line temporal calibration. By applying Time Delay Estimation on the tracking data, we show that the temporal offset between generic Mixed Reality spatial tracking sensors can be calibrated. To show the correctness and the feasibility of this approach, we have examined different variations of our method and evaluated various combinations of tracking sensors. We furthermore integrated this time synchronization method into our UBITRACK Mixed Reality tracking framework to provide facilities for calibration and real-time data alignment.

Comparison of 2D and 3D GUI Widgets for Stereoscopic Multitouch Setups

Recent developments in the area of interactive entertainment have suggested to combine stereoscopic visualization with multi-touch displays, which has the potential to open up new vistas for natural interaction with interactive three-dimensional (3D) applications. However, the question arises how the user interfaces for system control in such 3D setups should be designed in order to provide an effective user experience. In this article we introduce 3D GUI widgets for interaction with stereoscopic touch displays. The design of the widgets was inspired to skeuomorphism and affordances in such a way that the user should be able to operate the virtual objects in the same way as their real-world equivalents. We evaluate the developed widgets and compared them with their 2D counterparts in the scope of an example application in order to analyze the usability of and user behavior with the widgets. The results reveal differences in user behavior with and without stereoscopic display during touch interaction, and show that the developed 2D as well as 3D GUI widgets can be used effectively in different applications.

Semantic Methods for Intelligent Distributed Design Environments

Continuous advancements in technology have led to increasingly comprehensive and distributed product development processes while in pursuit of improved products at reduced costs. Information associated with these products is ever changing, and structured frameworks have become integral to managing such fluid information. Ontologies and the Semantic Web have emerged as key alternatives for capturing product knowledge in both a human-readable and computable manner. The primary and conclusive focus of this research is to characterize relationships formed within methodically developed distributed design knowledge frameworks to ultimately provide a pervasive real-time awareness in distributed design processes. Utilizing formal logics in the form of the Semantic Web’s OWL and SWRL, causal relationships are expressed to guide and facilitate knowledge acquisition as well as identify contradictions between knowledge in a knowledge base. To improve the efficiency during both the development and operational phases of these “intelligent” frameworks, a semantic relatedness algorithm is designed specifically to identify and rank underlying relationships within product development processes. After reviewing several semantic relatedness measures, three techniques, including a novel meronomic technique, are combined to create AIERO, the Algorithm for Identifying Engineering Relationships in Ontologies. In determining its applicability and accuracy, AIERO was applied to three separate, independently developed ontologies. The results indicate AIERO is capable of consistently returning relatedness values one would intuitively expect. To assess the effectiveness of AIERO in exposing underlying causal relationships across product development platforms, a case study involving the development of an industry-inspired printed circuit board (PCB) is presented. After instantiating the PCB knowledge base and developing an initial set of rules, FIDOE, the Framework for Intelligent Distributed Ontologies in Engineering, was employed to identify additional causal relationships through extensional relatedness measurements. In a conclusive PCB redesign, the resulting “intelligent” framework demonstrates its ability to pass values between instances, identify inconsistencies amongst instantiated knowledge, and identify conflicting values within product development frameworks. The results highlight how the introduced semantic methods can enhance the current knowledge acquisition, knowledge management, and knowledge validation capabilities of traditional knowledge bases.

New Technologies and Concepts for Rehabilitation in the Acute Phase of Stroke: A Collaborative Matrix

The process of developing a successful stroke rehabilitation methodology requires four key components: a good understanding of the pathophysiological mechanisms underlying this brain disease, clear neuroscientific hypotheses to guide therapy, adequate clinical assessments of its efficacy on multiple timescales, and a systematic approach to the application of modern technologies to assist in the everyday work of therapists. Achieving this goal requires collaboration between neuroscientists, technologists and clinicians to develop well-founded systems and clinical protocols that are able to provide quantitatively validated improvements in patient rehabilitation outcomes. In this article we present three new applications of complementary technologies developed in an interdisciplinary matrix for acute-phase upper limb stroke rehabilitation – functional electrical stimulation, arm robot-assisted therapy and virtual reality-based cognitive therapy. We also outline the neuroscientific basis of our approach, present our detailed clinical assessment protocol and provide preliminary results from patient testing of each of the three systems showing their viability for patient use.

Glucocorticoids enhance in vivo exposure-based therapy outcome in spider phobia

BACKGROUND: Preclinical and clinical studies indicate that the administration of glucocorticoids may promote fear extinction processes. In particular, it has been shown that glucocorticoids enhance virtual reality based exposure therapy of fear of heights. Here, we investigate whether glucocorticoids enhance the outcome of in vivo exposure-based group therapy of spider phobia. METHODS: In a double blind, block-randomized, placebo-controlled, between-subject study design, 22 patients with specific phobia of spiders were treated with two sessions of in vivo exposure-based group therapy. Cortisol (20 mg) or placebo was orally administered 1 hr before each therapy session. Patients returned for a follow-up assessment one month after therapy. RESULTS: Exposure-based group therapy led to a significant decrease in phobic symptoms as assessed with the Fear of Spiders Questionnaire (FSQ) from pretreatment to immediate posttreatment and to follow-up. The administration of cortisol to exposure therapy resulted in increased salivary cortisol concentrations and a significantly greater reduction in fear of spiders (FSQ) as compared to placebo at follow-up, but not immediately posttreatment. Furthermore, cortisol-treated patients reported significantly less anxiety during standardized exposure to living spiders at follow-up than placebo-treated subjects. Notably, groups did not differ in phobia-unrelated state-anxiety before and after the exposure sessions and at follow-up. CONCLUSIONS: These findings indicate that adding cortisol to in vivo exposure-based group therapy of spider phobia enhances treatment outcome.

Design and Test of a 0.5 THz Dipole Antenna With Integrated Schottky Diode Detector on a High Dielectric Constant Ceramic Electromagnetic Bandgap Substrate

This paper presents the first analysis of the input impedance and radiation properties of a dipole antenna, placed on top of Fan 's three-dimensional electromagnetic bandgap (EBG) structure, (Applied Physics Letters, 1994) constructed using a high dielectric constant ceramic. The best position of the dipole on the EBG surface is determined following impedance and radiation pattern analyses. Based on this optimum configuration an integrated Schottky heterodyne detector was designed, manufactured and tested from 0.48 to 0.52 THz. The main antenna features were not degraded by the high dielectric constant substrate due to the use of the EBG approach. Measured radiation patterns are in good agreement with the predicted ones.

Influence of termal fluctuations in radiation damage cascade production and defect dynamics in tungsten

The detailed study of the deterioration suffered by the materials of the components of a nuclear facility, in particular those forming part of the reactor core, is a topic of great interest which importance derives in large technological and economic implications. Since changes in the atomic-structural properties of relevant components pose a risk to the smooth operation with clear consequences for security and life of the plant, controlling these factors is essential in any development of engineering design and implementation. In recent times, tungsten has been proposed as a structural material based on its good resistance to radiation, but still needs to be done an extensive study on the influence of temperature on the behavior of this material under radiation damage. This work aims to contribute in this regard. Molecular Dynamics (MD) simulations were carried out to determine the influence of temperature fluctuations on radiation damage production and evolution in Tungsten. We have particularly focused our study in the dynamics of defect creation, recombination, and diffusion properties. PKA energies were sampled in a range from 5 to 50 KeV. Three different temperature scenarios were analyzed, from very low temperatures (0-200K), up to high temperature conditions (300-500 K). We studied the creation of defects, vacancies and interstitials, recombination rates, diffusion properties, cluster formation, their size and evolution. Simulations were performed using Lammps and the Zhou EAM potential for W