Marker-Based Embodied Interaction for Handheld Augmented Reality Games

This article deals with embodied user interfaces for handheld augmented reality games, which consist of both physical and virtual components. We have developed a number of spatial interaction techniques that optically capture the device's movement and orientation relative to a visual marker. Such physical interactions in 3-D space enable manipulative control of mobile games. In addition to acting as a physical controller that recognizes multiple game-dependent gestures, the mobile device augments the camera view with graphical overlays. We describe three game prototypes that use ubiquitous product packaging and other passive media as backgrounds for handheld augmentation. The prototypes can be realized on widely available off-the-shelf hardware and require only minimal setup and infrastructure support.

Evaluation of Binocular Eye Trackers and Algorithms for 3D Gaze Interaction in Virtual Reality Environments

Tracking user’s visual attention is a fundamental aspect in novel human-computer interaction paradigms found in Virtual Reality. For example, multimodal interfaces or dialogue-based communications with virtual and real agents greatly benefit from the analysis of the user’s visual attention as a vital source for deictic references or turn-taking signals. Current approaches to determine visual attention rely primarily on monocular eye trackers. Hence they are restricted to the interpretation of two-dimensional fixations relative to a defined area of projection. The study presented in this article compares precision, accuracy and application performance of two binocular eye tracking devices. Two algorithms are compared which derive depth information as required for visual attention-based 3D interfaces. This information is further applied to an improved VR selection task in which a binocular eye tracker and an adaptive neural network algorithm is used during the disambiguation of partly occluded objects.

Visual Fixation for 3D Video Stabilization

Visual fixation is employed by humans and some animals to keep a specific 3D location at the center of the visual gaze. Inspired by this phenomenon in nature, this paper explores the idea to transfer this mechanism to the context of video stabilization for a handheld video camera. A novel approach is presented that stabilizes a video by fixating on automatically extracted 3D target points. This approach is different from existing automatic solutions that stabilize the video by smoothing. To determine the 3D target points, the recorded scene is analyzed with a stateof- the-art structure-from-motion algorithm, which estimates camera motion and reconstructs a 3D point cloud of the static scene objects. Special algorithms are presented that search either virtual or real 3D target points, which back-project close to the center of the image for as long a period of time as possible. The stabilization algorithm then transforms the original images of the sequence so that these 3D target points are kept exactly in the center of the image, which, in case of real 3D target points, produces a perfectly stable result at the image center. Furthermore, different methods of additional user interaction are investigated. It is shown that the stabilization process can easily be controlled and that it can be combined with state-of-theart tracking techniques in order to obtain a powerful image stabilization tool. The approach is evaluated on a variety of videos taken with a hand-held camera in natural scenes.

Visual complaints and visual hallucinations in Parkinson's disease

Background Visual symptoms are common in Parkinson's disease (PD) and are frequently under-diagnosed. The detection of visual symptoms is important for differential diagnosis and patient management. Aim To establish the prevalence of recurrent visual complaints (RVC) and recurrent visual hallucinations (RVH) and to investigate their interaction in PD patients and controls. Methods This cross-sectional study included 88 PD patients and 90 controls. RVC and RVH were assessed with a visual symptom questionnaire and the North-East-Visual-Hallucinations-Interview (NEVHI). Results Double vision (PD vs. Controls: 18.2% vs. 1.3%; p < 0.001), misjudging objects when walking (PD vs. Controls: 12.5% vs. 1.3%; p < 0.01), words moving whilst reading (PD vs. Controls: 17.0% vs. 1.3%; p < 0.001) and freezing in narrow spaces (PD vs. Controls: 30.7% vs. 0%; p < 0.001) were almost exclusively found in PD patients. The same was true for recurrent complex visual hallucinations and illusions (PD vs. Controls: both 17.0% vs. 0%; p < 0.001). Multiple RVC (43.2% vs. 15.8%) and multiple RVH (29.5% vs. 5.6%) were also more common in PD patients (both p < 0.001). RVC did not predict recurrent complex visual hallucinations; but double vision (p = 0.018, R2 = 0.302) and misjudging objects (p = 0.002, R2 = 0.302) predicted passage hallucinations. Misjudging objects also predicted the feeling of presence (p = 0.010, R2 = 0.321). Conclusions Multiple and recurrent visual symptoms are common in PD. RVC emerged as risk factors predictive of the minor forms of hallucinations, but not recurrent complex visual hallucinations.

Effects of age and eccentricity on visual target detection

The aim of this study was to examine the effects of aging and target eccentricity on a visual search task comprising 30 images of everyday life projected into a hemisphere, realizing a ±90° visual field. The task performed binocularly allowed participants to freely move their eyes to scan images for an appearing target or distractor stimulus (presented at 10°; 30°, and 50° eccentricity). The distractor stimulus required no response, while the target stimulus required acknowledgment by pressing the response button. One hundred and seventeen healthy subjects (mean age = 49.63 years, SD = 17.40 years, age range 20–78 years) were studied. The results show that target detection performance decreases with age as well as with increasing eccentricity, especially for older subjects. Reaction time also increases with age and eccentricity, but in contrast to target detection, there is no interaction between age and eccentricity. Eye movement analysis showed that younger subjects exhibited a passive search strategy while older subjects exhibited an active search strategy probably as a compensation for their reduced peripheral detection performance.

Atypical visual processing in posttraumatic stress disorder

BACKGROUND: Many patients with Posttraumatic Stress Disorder (PTSD) feel overwhelmed in situations with high levels of sensory input, as in crowded situations with complex sensory characteristics. These difficulties might be related to subtle sensory processing deficits similar to those that have been found for sounds in electrophysiological studies. METHOD: Visual processing was investigated with functional magnetic resonance imaging in trauma-exposed participants with (N = 18) and without PTSD (N = 21) employing a picture-viewing task. RESULTS: Activity observed in response to visual scenes was lower in PTSD participants 1) in the ventral stream of the visual system, including striate and extrastriate, inferior temporal, and entorhinal cortices, and 2) in dorsal and ventral attention systems (P < 0.05, FWE-corrected). These effects could not be explained by the emotional salience of the pictures. CONCLUSIONS: Visual processing was substantially altered in PTSD in the ventral visual stream, a component of the visual system thought to be responsible for object property processing. Together with previous reports of subtle auditory deficits in PTSD, these findings provide strong support for potentially important sensory processing deficits, whose origins may be related to dysfunctional attention processes.

Multifactor effects and evidence of potential interaction between complement factor H Y402H and LOC387715 A69S in age-related macular degeneration.

BACKGROUND: Variants in the complement cascade genes and the LOC387715/HTRA1, have been widely reported to associate with age-related macular degeneration (AMD), the most common cause of visual impairment in industrialized countries. METHODS/PRINCIPAL FINDINGS: We investigated the association between the LOC387715 A69S and complement component C3 R102G risk alleles in the Finnish case-control material and found a significant association with both variants (OR 2.98, p = 3.75 x 10(-9); non-AMD controls and OR 2.79, p = 2.78 x 10(-19), blood donor controls and OR 1.83, p = 0.008; non-AMD controls and OR 1.39, p = 0.039; blood donor controls), respectively. Previously, we have shown a strong association between complement factor H (CFH) Y402H and AMD in the Finnish population. A carrier of at least one risk allele in each of the three susceptibility loci (LOC387715, C3, CFH) had an 18-fold risk of AMD when compared to a non-carrier homozygote in all three loci. A tentative gene-gene interaction between the two major AMD-associated loci, LOC387715 and CFH, was found in this study using a multiplicative (logistic regression) model, a synergy index (departure-from-additivity model) and the mutual information method (MI), suggesting that a common causative pathway may exist for these genes. Smoking (ever vs. never) exerted an extra risk for AMD, but somewhat surprisingly, only in connection with other factors such as sex and the C3 genotype. Population attributable risks (PAR) for the CFH, LOC387715 and C3 variants were 58.2%, 51.4% and 5.8%, respectively, the summary PAR for the three variants being 65.4%. CONCLUSIONS/SIGNIFICANCE: Evidence for gene-gene interaction between two major AMD associated loci CFH and LOC387715 was obtained using three methods, logistic regression, a synergy index and the mutual information (MI) index.

Effects of anxiety on decision making and visual search behaviour in complex sport situations

Introduction Current empirical findings indicate that the efficiency of decision making (both for experts and near-experts) in simple situations is reduced under increased stress (Wilson, 2008). Explaining the phenomenon, the Attentional Control Theory (ACT, Eysenck et al., 2007) postulates an impairment of attentional processes resulting in a less efficient processing of visual information. From a practitioner’s perspective, it would be highly relevant to know whether this phenomenon can also be found in complex sport situations like in the game of football. Consequently, in the present study, decision making of football players was examined under regular vs. increased anxiety conditions. Methods 22 participants (11 experts and 11 near-experts) viewed 24 complex football situations (counterbalanced) in two anxiety conditions from the perspective of the last defender. They had to decide as fast and accurate as possible on the next action of the player in possession (options: shot on goal, dribble or pass to a designated team member) for equal numbers of trials in a near and far distance condition (based on the position of the player in possession). Anxiety was manipulated via a competitive environment, false feedback as well as ego threats. Decision time and accuracy, gaze behaviour (e.g., fixation duration on different locations) as well as state anxiety and mental effort were used as dependent variables and analysed with 2 (expertise) x 2 (distance) x 2 (anxiety) ANOVAs with repeated measures on the last two factors. Besides expertise differences, it was hypothesised that, based on ACT, increased anxiety reduces performance efficiency and impairs gaze behaviour. Results and Discussion Anxiety was manipulated successfully, indicated by higher ratings of state anxiety, F(1, 20) = 13.13, p < .01, ηp2 = .40. Besides expertise differences in decision making – experts responded faster, F(1, 20) = 11.32, p < .01, ηp2 = .36, and more accurate, F(1,20) = 23.93, p < .01, ηp2 = .55, than near-experts – decision time, F(1, 20) = 9.29, p < .01, ηp2 = .32, and mental effort, F(1, 20) = 7.33, p = .01, ηp2 = .27, increased for both groups in the high anxiety condition. This result confirms the ACT assumption that processing efficiency is reduced when being anxious. Replicating earlier findings, a significant expertise by distance interaction could be observed, F(1, 18) = 18.53, p < .01, ηp2 = .51), with experts fixating longer on the player in possession or the ball in the near distance and longer on other opponents, teammates and free space in the far distance condition. This shows that experts are able to adjust their gaze behaviour to affordances of displayed playing patterns. Additionally, a three way interaction was found, F(1, 18) = 7.37 p = .01, ηp2 = .29, revealing that experts utilised a reduced number of fixations in the far distance condition when being anxious indicating a reduced ability to pick up visual information. Since especially the visual search behaviour of experts was impaired, the ACT prediction that particularly top-down processes are affected by anxiety could be confirmed. Taken together, the results show that sports performance is negatively influenced by anxiety since longer response times, higher mental effort and inefficient visual search behaviour were observed. From a practitioner’s perspective, this finding might suggest preferring (implicit) perceptual cognitive training; however, this recommendation needs to be empirically supported in intervention studies. References: Eysenck, M. W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. Emotion, 7, 336-353. Wilson, M. (2008). From processing efficiency to attentional control: A mechanistic account of the anxiety-performance relationship. Int. Review of Sport and Exercise Psychology, 1, 184-201.

Visual hallucinations in the psychosis spectrum and comparative information from neurodegenerative disorders and eye disease

Much of the research on visual hallucinations (VHs) has been conducted in the context of eye disease and neurodegenerative conditions, but little is known about these phenomena in psychiatric and nonclinical populations. The purpose of this article is to bring together current knowledge regarding VHs in the psychosis phenotype and contrast this data with the literature drawn from neurodegenerative disorders and eye disease. The evidence challenges the traditional views that VHs are atypical or uncommon in psychosis. The weighted mean for VHs is 27% in schizophrenia, 15% in affective psychosis, and 7.3% in the general community. VHs are linked to a more severe psychopathological profile and less favorable outcome in psychosis and neurodegenerative conditions. VHs typically co-occur with auditory hallucinations, suggesting a common etiological cause. VHs in psychosis are also remarkably complex, negative in content, and are interpreted to have personal relevance. The cognitive mechanisms of VHs in psychosis have rarely been investigated, but existing studies point to source-monitoring deficits and distortions in top-down mechanisms, although evidence for visual processing deficits, which feature strongly in the organic literature, is lacking. Brain imaging studies point to the activation of visual cortex during hallucinations on a background of structural and connectivity changes within wider brain networks. The relationship between VHs in psychosis, eye disease, and neurodegeneration remains unclear, although the pattern of similarities and differences described in this review suggests that comparative studies may have potentially important clinical and theoretical implications.

A new method to measure higher visual functions in an immersive environment

BACKGROUND: Higher visual functions can be defined as cognitive processes responsible for object recognition, color and shape perception, and motion detection. People with impaired higher visual functions after unilateral brain lesion are often tested with paper pencil tests, but such tests do not assess the degree of interaction between the healthy brain hemisphere and the impaired one. Hence, visual functions are not tested separately in the contralesional and ipsilesional visual hemifields. METHODS: A new measurement setup, that involves real-time comparisons of shape and size of objects, orientation of lines, speed and direction of moving patterns, in the right or left visual hemifield, has been developed. The setup was implemented in an immersive environment like a hemisphere to take into account the effects of peripheral and central vision, and eventual visual field losses. Due to the non-flat screen of the hemisphere, a distortion algorithm was needed to adapt the projected images to the surface. Several approaches were studied and, based on a comparison between projected images and original ones, the best one was used for the implementation of the test. Fifty-seven healthy volunteers were then tested in a pilot study. A Satisfaction Questionnaire was used to assess the usability of the new measurement setup. RESULTS: The results of the distortion algorithm showed a structural similarity between the warped images and the original ones higher than 97%. The results of the pilot study showed an accuracy in comparing images in the two visual hemifields of 0.18 visual degrees and 0.19 visual degrees for size and shape discrimination, respectively, 2.56° for line orientation, 0.33 visual degrees/s for speed perception and 7.41° for recognition of motion direction. The outcome of the Satisfaction Questionnaire showed a high acceptance of the battery by the participants. CONCLUSIONS: A new method to measure higher visual functions in an immersive environment was presented. The study focused on the usability of the developed battery rather than the performance at the visual tasks. A battery of five subtasks to study the perception of size, shape, orientation, speed and motion direction was developed. The test setup is now ready to be tested in neurological patients.

Mechanisms of Ligand–Protein Interaction in Sec-14-like Transporters Investigated by Computer Simulations

We review our recent work on protein-ligand interactions in vitamin transporters of the Sec-14-like protein. Our studies focused on the cellular-retinaldehyde binding protein (CRALBP) and the alpha-tocopherol transfer protein (alpha-TTP). CRALBP is responsible for mobilisation and photo-protection of short-chain cis-retinoids in the dim-light visual cycle or rod photoreceptors. alpha-TTP is a key protein responsible for selection and retention of RRR-alpha-tocopherol, the most active isoform of vitamin E in superior animals. Our simulation studies evidence how subtle chemical variations in the substrate can lead to significant distortion in the structure of the complex, and how these changes can either lead to new protein function, or be used to model engineered protein variants with tailored properties. Finally, we show how integration of computational and experimental results can contribute in synergy to the understanding of fundamental processes at the biomolecular scale.

Perception of co-speech gestures in aphasic patients: A visual exploration study during the observation of dyadic conversations.

BACKGROUND Co-speech gestures are part of nonverbal communication during conversations. They either support the verbal message or provide the interlocutor with additional information. Furthermore, they prompt as nonverbal cues the cooperative process of turn taking. In the present study, we investigated the influence of co-speech gestures on the perception of dyadic dialogue in aphasic patients. In particular, we analysed the impact of co-speech gestures on gaze direction (towards speaker or listener) and fixation of body parts. We hypothesized that aphasic patients, who are restricted in verbal comprehension, adapt their visual exploration strategies. METHODS Sixteen aphasic patients and 23 healthy control subjects participated in the study. Visual exploration behaviour was measured by means of a contact-free infrared eye-tracker while subjects were watching videos depicting spontaneous dialogues between two individuals. Cumulative fixation duration and mean fixation duration were calculated for the factors co-speech gesture (present and absent), gaze direction (to the speaker or to the listener), and region of interest (ROI), including hands, face, and body. RESULTS Both aphasic patients and healthy controls mainly fixated the speaker's face. We found a significant co-speech gesture × ROI interaction, indicating that the presence of a co-speech gesture encouraged subjects to look at the speaker. Further, there was a significant gaze direction × ROI × group interaction revealing that aphasic patients showed reduced cumulative fixation duration on the speaker's face compared to healthy controls. CONCLUSION Co-speech gestures guide the observer's attention towards the speaker, the source of semantic input. It is discussed whether an underlying semantic processing deficit or a deficit to integrate audio-visual information may cause aphasic patients to explore less the speaker's face.

Effects of (un)predictable whole body vibrations on visual performance

Over recent years, it has repeatedly been shown that optimal gaze strategies enhance motor control (e.g., Foulsham, 2015). However, little is known, whether, vice versa, visual performance can be improved by optimized motor control. Consequently, in two studies, we investigated visual performance as a function of motor control strategies and task parameters, respectively. In Experiment 1, 72 participants were tested on visual acuity (Landolt) and contrast sensitivity (Grating), while standing in two different postures (upright vs. squat) on a ZEPTOR-platform that vibrated at four different frequencies (0, 4, 8, 12 Hz). After each test, perceived exertion (Borg) was assessed. Significant interactions were revealed for both tests, Landolt: F(3,213)=13.25, p<.01, ηp2=.16, Grating: F(3,213)=4.27, p<.01, ηp2=.06, elucidating a larger loss of acuity/contrast sensitivity with increasing frequencies for the upright compared with the squat posture. For perceived exertion, however, a diametrical interaction for frequency was found for acuity, F(3,213)=7.45, p<.01, ηp2=.09, and contrast sensitivity, F(3,213)=7.08, p < .01, ηp2=.09, substantiating that the impaired visual performance cannot be attributed to exertion. Consequently, the squat posture could permit better head and, hence, gaze stabilization. In Experiment 2, 64 participants performed the same tests while standing in a squat position on a ski-simulator, which vibrated with two different frequencies (2.4, 3.6 Hz) and amplitudes (50, 100 mm) in a predictable or unpredictable manner. Control strategies were identified by tracking segmental motion, which allows to derive damping characteristics. Considerable main effects were found for frequency, all F’s(1,52)>10.31, all p’s<.01, all ηp2’s>.16, as well as, in the acuity test, for predictability, F(1,52)=10.31, p<.01, ηp2=.17, and by tendency for amplitude, F(1,52)=3.53, p=.06, ηp2=.06. A significant correlation between the damping amplitude in the knee joint and the performance drop in visual acuity, r=-.97, p<.001, again points towards the importance of motor control strategies to maintain optimal visual performance.

Perception of co-speech gestures in aphasic patients: A visual exploration study during the observation of dyadic conversations

Background: Co-speech gestures are part of nonverbal communication during conversations. They either support the verbal message or provide the interlocutor with additional information. Furthermore, they prompt as nonverbal cues the cooperative process of turn taking. In the present study, we investigated the influence of co-speech gestures on the perception of dyadic dialogue in aphasic patients. In particular, we analysed the impact of co-speech gestures on gaze direction (towards speaker or listener) and fixation of body parts. We hypothesized that aphasic patients, who are restricted in verbal comprehension, adapt their visual exploration strategies. Methods: Sixteen aphasic patients and 23 healthy control subjects participated in the study. Visual exploration behaviour was measured by means of a contact-free infrared eye-tracker while subjects were watching videos depicting spontaneous dialogues between two individuals. Cumulative fixation duration and mean fixation duration were calculated for the factors co-speech gesture (present and absent), gaze direction (to the speaker or to the listener), and region of interest (ROI), including hands, face, and body. Results: Both aphasic patients and healthy controls mainly fixated the speaker’s face. We found a significant co-speech gesture x ROI interaction, indicating that the presence of a co-speech gesture encouraged subjects to look at the speaker. Further, there was a significant gaze direction x ROI x group interaction revealing that aphasic patients showed reduced cumulative fixation duration on the speaker’s face compared to healthy controls. Conclusion: Co-speech gestures guide the observer’s attention towards the speaker, the source of semantic input. It is discussed whether an underlying semantic processing deficit or a deficit to integrate audio-visual information may cause aphasic patients to explore less the speaker’s face. Keywords: Gestures, visual exploration, dialogue, aphasia, apraxia, eye movements

Dynamic population coding in primary visual cortex

More than a century ago Ramon y Cajal pioneered the description of neural circuits. Currently, new techniques are being developed to streamline the characterization of entire neural circuits. Even if this 'connectome' approach is successful, it will represent only a static description of neural circuits. Thus, a fundamental question in neuroscience is to understand how information is dynamically represented by neural populations. In this thesis, I studied two main aspects of dynamical population codes. ^ First, I studied how the exposure or adaptation, for a fraction of a second to oriented gratings dynamically changes the population response of primary visual cortex neurons. The effects of adaptation to oriented gratings have been extensively explored in psychophysical and electrophysiological experiments. However, whether rapid adaptation might induce a change in the primary visual cortex's functional connectivity to dynamically impact the population coding accuracy is currently unknown. To address this issue, we performed multi-electrode recordings in primary visual cortex, where adaptation has been previously shown to induce changes in the selectivity and response amplitude of individual neurons. We found that adaptation improves the population coding accuracy. The improvement was more prominent for iso- and orthogonal orientation adaptation, consistent with previously reported psychophysical experiments. We propose that selective decorrelation is a metabolically inexpensive mechanism that the visual system employs to dynamically adapt the neural responses to the statistics of the input stimuli to improve coding efficiency. ^ Second, I investigated how ongoing activity modulates orientation coding in single neurons, neural populations and behavior. Cortical networks are never silent even in the absence of external stimulation. The ongoing activity can account for up to 80% of the metabolic energy consumed by the brain. Thus, a fundamental question is to understand the functional role of ongoing activity and its impact on neural computations. I studied how the orientation coding by individual neurons and cell populations in primary visual cortex depend on the spontaneous activity before stimulus presentation. We hypothesized that since the ongoing activity of nearby neurons is strongly correlated, it would influence the ability of the entire population of orientation-selective cells to process orientation depending on the prestimulus spontaneous state. Our findings demonstrate that ongoing activity dynamically filters incoming stimuli to shape the accuracy of orientation coding by individual neurons and cell populations and this interaction affects behavioral performance. In summary, this thesis is a contribution to the study of how dynamic internal states such as rapid adaptation and ongoing activity modulate the population code accuracy. ^