Supporting collocated and at-a-distance experiences with TV and VR displays

Televisions (TVs) and VR Head-Mounted Displays (VR HMDs) are used in shared and social spaces in the home. This thesis posits that these displays do not sufficiently reflect the collocated, social contexts in which they reside, nor do they sufficiently support shared experiences at-a-distance. This thesis explores how the role of TVs and VR HMDs can go beyond presenting a single entertainment experience, instead supporting social and shared use in both collocated and at-a-distance contexts. For collocated TV, this thesis demonstrates that the TV can be augmented to facilitate multi-user interaction, support shared and independent activities and multi-user use through multi-view display technology, and provide awareness of the multi-screen activity of those in the room, allowing the TV to reflect the social context in which it resides. For at-a-distance TV, existing smart TVs are shown to be capable of supporting synchronous at-a-distance activity, broadening the scope of media consumption beyond the four walls of the home. For VR HMDs, collocated proximate persons can be seamlessly brought into mixed reality VR experiences based on engagement, improving VR HMD usability. Applied to at-a-distance interactions, these shared mixed reality VR experiences can enable more immersive social experiences that approximate viewing together as if in person, compared to at-a-distance TV. Through an examination of TVs and VR HMDs, this thesis demonstrates that consumer display technology can better support users to interact, and share experiences and activities, with those they are close to.

Which way is up? Grounded mental representations of space

Processing language is postulated to involve a mental simulation, or re-enactment of perceptual, motor, and introspective states that were acquired experientially (Barsalou, 1999, 2008). One such aspect that is mentally simulated during processing of certain concepts is spatial location. For example, upon processing the word “moon” the prominent spatial location of the concept (e.g. ‘upward’) is mentally simulated. In six eye-tracking experiments, we investigate how mental simulations of spatial location affect processing. We first address a conflict in previous literature whereby processing is shown to be impacted in both a facilitatory and inhibitory way. Two of our experiments showed that mental simulations of spatial association facilitate saccades launched toward compatible locations; however, a third experiment showed an inhibitory effect on saccades launched towards incompatible locations. We investigated these differences with further experiments, which led us to conclude that the nature of the effect (facilitatory or inhibitory) is dependent on the demands of the task and, in fitting with the theory of Grounded Cognition (Barsalou, 2008), that mental simulations impact processing in a dynamic way. Three further experiments explored the nature of verticality – specifically, whether ‘up’ is perceived as away from gravity, or above our head. Using similar eye-tracking methods, and by manipulating the position of participants, we were able to dissociate these two possible standpoints. The results showed that mental simulations of spatial location facilitated saccades to compatible locations, but only when verticality was dissociated from gravity (i.e. ‘up’ was above the participant’s head). We conclude that this is not due to an ‘embodied’ mental simulation, but rather a result of heavily ingrained visuo-motor association between vertical space and eye movements.