Increasing Realism and Supporting Content Planning for Dynamic Scenes in a Mixed Reality System incorporating a Time-of-Flight Camera

For broadcasting purposes MIXED REALITY, the combination of real and virtual scene content, has become ubiquitous nowadays. Mixed Reality recording still requires expensive studio setups and is often limited to simple color keying. We present a system for Mixed Reality applications which uses depth keying and provides threedimensional mixing of real and artificial content. It features enhanced realism through automatic shadow computation which we consider a core issue to obtain realism and a convincing visual perception, besides the correct alignment of the two modalities and correct occlusion handling. Furthermore we present a possibility to support placement of virtual content in the scene. Core feature of our system is the incorporation of a TIME-OF-FLIGHT (TOF)-camera device. This device delivers real-time depth images of the environment at a reasonable resolution and quality. This camera is used to build a static environment model and it also allows correct handling of mutual occlusions between real and virtual content, shadow computation and enhanced content planning. The presented system is inexpensive, compact, mobile, flexible and provides convenient calibration procedures. Chroma-keying is replaced by depth-keying which is efficiently performed on the GRAPHICS PROCESSING UNIT (GPU) by the usage of an environment model and the current ToF-camera image. Automatic extraction and tracking of dynamic scene content is herewith performed and this information is used for planning and alignment of virtual content. An additional sustainable feature is that depth maps of the mixed content are available in real-time, which makes the approach suitable for future 3DTV productions. The presented paper gives an overview of the whole system approach including camera calibration, environment model generation, real-time keying and mixing of virtual and real content, shadowing for virtual content and dynamic object tracking for content planning.

Connecting Interactive Arts and Virtual Reality with Enaction

This paper reports on a Virtual Reality theater experiment named Il était Xn fois, conducted by artists and computer scientists working in cognitive science. It offered the opportunity for knowledge and ideas exchange between these groups, highlighting the benefits of collaboration of this kind. Section 1 explains the link between enaction in cognitive science and virtual reality, and specifically the need to develop an autonomous entity which enhances presence in an artificial world. Section 2 argues that enactive artificial intelligence is able to produce such autonomy. This was demonstrated by the theatrical experiment, "Il était Xn fois" (in English: Once upon Xn time), explained in section 3. Its first public performance was in 2009, by the company Dérézo. The last section offers the view that enaction can form a common ground between the artistic and computer science areas.

Virtual Reality as a Support Tool for Ergonomic-Style Convergence

The competitive industrial context compels companies to speed-up every new product design. In order to keep designing products that meet the needs of the end user, a human centered concurrent product design methodology has been proposed. Its setting up is complicated by the difficulties of collaboration between experts involved inthe design process. In order to ease this collaboration, we propose the use of virtual reality as an intermediate design representation in the form of light and specialized immersive convergence support applications. In this paper, we present the As Soon As Possible (ASAP) methodology making possible the development of these tools while ensuring their usefulness and usability. The relevance oft his approach is validated by an industrial use case through the design of an ergonomic-style convergence support tool.

Simulating Wind and Warmth in Virtual Reality: Conception, Realization and Evaluation for a CAVE Environment

Wind and warmth sensations proved to be able to enhance users' state of presence in Virtual Reality applications. Still, only few projects deal with their detailed effect on the user and general ways of implementing such stimuli. This work tries to fill this gap: After analyzing requirements for hardware and software concerning wind and warmth simulations, a hardware and also a software setup for the application in a CAVE environment is proposed. The setup is evaluated with regard to technical details and requirements, but also - in the form of a pilot study - in view of user experience and presence. Our setup proved to comply with the requirements and leads to satisfactory results. To our knowledge, the low cost simulation system (approx. 2200 Euro) presented here is one of the most extensive, most flexible and best evaluated systems for creating wind and warmth stimuli in CAVE-based VR applications.

Influence of Information and Instructions on Human Behavior in Tunnel Accidents: A Virtual Reality Study

Human behavior is a major factor modulating the consequences of road tunnel accidents. We investigated the effect of information and instruction on drivers' behavior as well as the usability of virtual environments to simulate such emergency situations. Tunnel safety knowledge of the general population was assessed using an online questionnaire, and tunnel safety behavior was investigated in a virtual reality experiment. Forty-four participants completed three drives through a virtual road tunnel and were confronted with a traffic jam, no event, and an accident blocking the road. Participants were randomly assigned to a control group (no intervention), an informed group who read a brochure containing safety information prior to the tunnel drives, or an informed and instructed group who read the same brochure and received additional instructions during the emergency situation. Informed participants showed better and quicker safety behavior than the control group. Self-reports of anxiety were assessed three times during each drive. Anxiety was elevated during and after the emergency situation. The findings demonstrate problematic safety behavior in the control group and that knowledge of safety information fosters adequate behavior in tunnel emergencies. Enhanced anxiety ratings during the emergency situation indicate external validity of the virtual environment.

Cement-augmented anterior screw fixation of Type II odontoid fractures in elderly patients with osteoporosis

BACKGROUND CONTEXT: Closed reduction and internal fixation by an anterior approach is an established option for operative treatment of displaced Type II odontoid fractures. In elderly patients, however, inadequate screw purchase in osteoporotic bone can result in severe procedure-related complications. PURPOSE: To improve the stability of odontoid fracture screw fixation in the elderly using a new technique that includes injection of polymethylmethacrylat (PMMA) cement into the C2 body. STUDY DESIGN: Retrospective review of hospital and outpatient records as well as radiographs of elderly patients treated in a university hospital department of orthopedic surgery. PATIENT SAMPLE: Twenty-four elderly patients (8 males and 16 females; mean age, 81 years; range, 62-98 years) with Type II fractures of the dens. OUTCOME MEASURES: Complications, cement leakage (symptomatic/asymptomatic), operation time, loss of reduction, pseudarthrosis and revision surgery, patient complaints, return to normal activities, and signs of neurologic complications were all documented. METHODS: After closed reduction and anterior approach to the inferior border of C2, a guide wire is advanced to the tip of the odontoid under biplanar fluoroscopic control. Before the insertion of one cannulated, self-drilling, short thread screws, a 12 gauge Yamshidi cannula is inserted from anterior and 1 to 3 mL of high-viscosity PMMA cement is injected into the anteroinferior portion of the C2 body. During polymerization of the cement, the screws are further inserted using a lag-screw compression technique. The cervical spine then is immobilized with a soft collar for 8 weeks postoperatively. RESULTS: Anatomical reduction of the dens was achieved in all 24 patients. Mean operative time was 64 minutes (40-90 minutes). Early loss of reduction occurred in three patients, but revision surgery was indicated in only one patient 2 days after primary surgery. One patient died within the first eight postoperative weeks, one within 3 months after surgery. In five patients, asymptomatic cement leakage was observed (into the C1-C2 joint in three patients, into the fracture in two). Conventional radiologic follow-up at 2 and 6 months confirmed anatomical healing in 16 of the19 patients with complete follow-up. In two patients, the fractures healed in slight dorsal angulation; one patient developed a asymptomatic pseudarthrosis. All patients were able to resume their pretrauma level of activity. CONCLUSIONS: Cement augmentation of the screw in Type II odontoid fractures in elderly patients is technically feasible in a clinical setting with a low complication rate. This technique may improve screw purchase, especially in the osteoporotic C2 body.

Osteocyte lacunar density and area in newly formed bone of the augmented sinus

OBJECTIVES Osteocytes, the most common cells of the bone, are buried in lacunae. Density and area of the osteocyte lacunae change with increasing maturation of the newly formed bone. Evaluation of osteocyte lacunae can therefore provide insights into the process of graft consolidation. MATERIALS AND METHODS Here, we determined the osteocyte lacunar density (number of osteocyte lacunae per bone area; N.Ot/BAr) and the osteocyte lacunar area in μm(2) (Lac.Ar) in histological specimens 6 and 12 weeks after the sinuses of 10 minipigs were augmented with Bio-Oss(®) , a deproteinized bovine bone mineral, and Ostim(®) , an aqueous paste of synthetic nanoparticular hydroxyapatite. The region of interest was defined by the following criteria: (i) >1 mm from the host bone, (ii) >0.5 mm from the sinus mucosa, (iii) minimum area of 0.2 mm(2) , and (iv) bone tissue spanning at least two bone substitute particles. RESULTS The overall osteocyte lacunar density was significantly higher in the Bio-Oss(®) group than in the Ostim(®) group and decreased during the observation period at a similar range in both groups. The osteocyte lacunar area was smaller in the Bio-Oss(®) group than the Ostim(®) group but there was no significant change within the groups over time. CONCLUSIONS These results suggest that bone substitutes affect the osteocyte lacunar density and the osteocyte lacunar area in the newly formed bone within the augmented sinus in this particular model situation. These measures can provide insights into the maturation of newly formed bone in the augmented sinus.

Long-term clinical, microbiological, and radiographic outcomes of Brånemark™ implants installed in augmented maxillary bone for fixed full-arch rehabilitation

PURPOSE The purpose of this study was to document the long-term outcome of Brånemark implants installed in augmented maxillary bone and to identify parameters that are associated with peri-implant bone level. MATERIAL AND METHODS Patients of a periodontal practice who had been referred to a maxillofacial surgeon for iliac crest bone grafting in the atrophic maxilla were retrospectively recruited. Five months following grafting, they received 7-8 turned Brånemark implants. Following submerged healing of another 5 months, implants were uncovered and restorative procedures for fixed rehabilitation were initiated 2-3 months thereafter. The primary outcome variable was bone level defined as the distance from the implant-abutment interface to the first visible bone-to-implant contact. Secondary outcome variables included plaque index, bleeding index, probing depth, and levels of 40 species in subgingival plaque samples as identified by means of checkerboard DNA-DNA hybridization. RESULTS Nine out of 16 patients (eight females, one male; mean age 59) with 71 implants agreed to come in for evaluation after on average 9 years (SD 4; range 3-13) of function. One implant was deemed mobile at the time of inspection. Clinical conditions were acceptable with 11% of the implants showing pockets ≥ 5 mm. Periodontopathogens were encountered frequently and in high numbers. Clinical parameters and bacterial levels were highly patient dependent. The mean bone level was 2.30 mm (SD 1.53; range 0.00-6.95), with 23% of the implants demonstrating advanced resorption (bone level > 3 mm). Regression analysis showed a significant association of the patient (p < .001) and plaque index (p = .007) with bone level. CONCLUSIONS The long-term outcome of Brånemark implants installed in iliac crest-augmented maxillary bone is acceptable; however, advanced peri-implant bone loss is rather common and indicative of graft resorption. This phenomenon is patient dependent and seems also associated with oral hygiene.

Ecological validity of virtual reality daily living activities screening for early dementia: longitudinal study

Background: Dementia is a multifaceted disorder that impairs cognitive functions, such as memory, language, and executive functions necessary to plan, organize, and prioritize tasks required for goal-directed behaviors. In most cases, individuals with dementia experience difficulties interacting with physical and social environments. The purpose of this study was to establish ecological validity and initial construct validity of a fire evacuation Virtual Reality Day-Out Task (VR-DOT) environment based on performance profiles as a screening tool for early dementia. Objective: The objectives were (1) to examine the relationships among the performances of 3 groups of participants in the VR-DOT and traditional neuropsychological tests employed to assess executive functions, and (2) to compare the performance of participants with mild Alzheimer’s-type dementia (AD) to those with amnestic single-domain mild cognitive impairment (MCI) and healthy controls in the VR-DOT and traditional neuropsychological tests used to assess executive functions. We hypothesized that the 2 cognitively impaired groups would have distinct performance profiles and show significantly impaired independent functioning in ADL compared to the healthy controls. Methods: The study population included 3 groups: 72 healthy control elderly participants, 65 amnestic MCI participants, and 68 mild AD participants. A natural user interface framework based on a fire evacuation VR-DOT environment was used for assessing physical and cognitive abilities of seniors over 3 years. VR-DOT focuses on the subtle errors and patterns in performing everyday activities and has the advantage of not depending on a subjective rating of an individual person. We further assessed functional capacity by both neuropsychological tests (including measures of attention, memory, working memory, executive functions, language, and depression). We also evaluated performance in finger tapping, grip strength, stride length, gait speed, and chair stands separately and while performing VR-DOTs in order to correlate performance in these measures with VR-DOTs because performance while navigating a virtual environment is a valid and reliable indicator of cognitive decline in elderly persons. Results: The mild AD group was more impaired than the amnestic MCI group, and both were more impaired than healthy controls. The novel VR-DOT functional index correlated strongly with standard cognitive and functional measurements, such as mini-mental state examination (MMSE; rho=0.26, P=.01) and Bristol Activities of Daily Living (ADL) scale scores (rho=0.32, P=.001). Conclusions: Functional impairment is a defining characteristic of predementia and is partly dependent on the degree of cognitive impairment. The novel virtual reality measures of functional ability seem more sensitive to functional impairment than qualitative measures in predementia, thus accurately differentiating from healthy controls. We conclude that VR-DOT is an effective tool for discriminating predementia and mild AD from controls by detecting differences in terms of errors, omissions, and perseverations while measuring ADL functional ability.

An experimentally validated finite element method for augmented vertebral bodies

Background Finite element models of augmented vertebral bodies require a realistic modelling of the cement infiltrated region. Most methods published so far used idealized cement shapes or oversimplified material models for the augmented region. In this study, an improved, anatomy-specific, homogenized finite element method was developed and validated to predict the apparent as well as the local mechanical behavior of augmented vertebral bodies. Methods Forty-nine human vertebral body sections were prepared by removing the cortical endplates and scanned with high-resolution peripheral quantitative CT before and after injection of a standard and a low-modulus bone cement. Forty-one specimens were tested in compression to measure stiffness, strength and contact pressure distributions between specimens and loading-plates. From the remaining eight, fourteen cylindrical specimens were extracted from the augmented region and tested in compression to obtain material properties. Anatomy-specific finite element models were generated from the CT data. The models featured element-specific, density-fabric-based material properties, damage accumulation, real cement distributions and experimentally determined material properties for the augmented region. Apparent stiffness and strength as well as contact pressure distributions at the loading plates were compared between simulations and experiments. Findings The finite element models were able to predict apparent stiffness (R2 > 0.86) and apparent strength (R2 > 0.92) very well. Also, the numerically obtained pressure distributions were in reasonable quantitative (R2 > 0.48) and qualitative agreement with the experiments. Interpretation The proposed finite element models have proven to be an accurate tool for studying the apparent as well as the local mechanical behavior of augmented vertebral bodies.

Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery

BACKGROUND Stereotactic navigation technology can enhance guidance during surgery and enable the precise reproduction of planned surgical strategies. Currently, specific systems (such as the CAS-One system) are available for instrument guidance in open liver surgery. This study aims to evaluate the implementation of such a system for the targeting of hepatic tumors during robotic liver surgery. MATERIAL AND METHODS Optical tracking references were attached to one of the robotic instruments and to the robotic endoscopic camera. After instrument and video calibration and patient-to-image registration, a virtual model of the tracked instrument and the available three-dimensional images of the liver were displayed directly within the robotic console, superimposed onto the endoscopic video image. An additional superimposed targeting viewer allowed for the visualization of the target tumor, relative to the tip of the instrument, for an assessment of the distance between the tumor and the tool for the realization of safe resection margins. RESULTS Two cirrhotic patients underwent robotic navigated atypical hepatic resections for hepatocellular carcinoma. The augmented endoscopic view allowed for the definition of an accurate resection margin around the tumor. The overlay of reconstructed three-dimensional models was also used during parenchymal transection for the identification of vascular and biliary structures. Operative times were 240 min in the first case and 300 min in the second. There were no intraoperative complications. CONCLUSIONS The da Vinci Surgical System provided an excellent platform for image-guided liver surgery with a stable optic and instrumentation. Robotic image guidance might improve the surgeon's orientation during the operation and increase accuracy in tumor resection. Further developments of this technological combination are needed to deal with organ deformation during surgery.

Using virtual reality to teach special populations how to cope in crisis: the case of a virtual earthquake

The unique characteristics of special populations such as pre-school children and Down syndrome kids in crisis and their distorted self-image were never studied before, because of the difficulty of crisis reproduction. This study proposes a VR setting that tries to model some special population's behaviour in the time of crises and offers them a training scenario. The sample population consisted of 30 pre-school children and 20 children with Down syndrome. The VR setting involved a high-speed PC, a VPL EyePhone 1, a MR toolkit, a vibrations plate, a motion capture system and other sensors. The system measured and modelled the typical behaviour of these special populations in a Virtual Earthquake scenario with sight and sound and calculated a VR anthropomorphic model that reproduced their behaviour and emotional state. Afterwards one group received an emotionally enhanced VR self-image as feedback for their training, one group received a plain VR self-image and another group received verbal instructions. The findings strongly suggest that the training was a lot more biased by the emotionally enhanced VR self-image than the other approaches. These findings could highlight the special role of the self-image to therapy and training and the interesting role of imagination to emotions, motives and learning. Further studies could be done with various scenarios in order to measure the best-biased behaviour and establish the most natural and affective VR model. This presentation is going to highlight the main findings and some theories behind them.

Translating Rhetoric to Reality: The Future of Family and Children's Services

These remarks were first prepared by the author for the inauguration of the Marion Elizabeth Blue Endowed Professorship in Children and Families at the University of Michigan School of Social Work. They were delivered on October 5, 1999, and originally appeared as a monograph published by the University of Michigan School of Social Work in December 1999. They are reprinted here by permission.