Quantifying epistemic actions in human-computer interaction

As digital systems move away from traditional desktop setups, new interaction paradigms are emerging that better integrate with users’ realworld surroundings, and better support users’ individual needs. While promising, these modern interaction paradigms also present new challenges, such as a lack of paradigm-specific tools to systematically evaluate and fully understand their use. This dissertation tackles this issue by framing empirical studies of three novel digital systems in embodied cognition – an exciting new perspective in cognitive science where the body and its interactions with the physical world take a central role in human cognition. This is achieved by first, focusing the design of all these systems on a contemporary interaction paradigm that emphasizes physical interaction on tangible interaction, a contemporary interaction paradigm; and second, by comprehensively studying user performance in these systems through a set of novel performance metrics grounded on epistemic actions, a relatively well established and studied construct in the literature on embodied cognition. The first system presented in this dissertation is an augmented Four-in-a-row board game. Three different versions of the game were developed, based on three different interaction paradigms (tangible, touch and mouse), and a repeated measures study involving 36 participants measured the occurrence of three simple epistemic actions across these three interfaces. The results highlight the relevance of epistemic actions in such a task and suggest that the different interaction paradigms afford instantiation of these actions in different ways. Additionally, the tangible version of the system supports the most rapid execution of these actions, providing novel quantitative insights into the real benefits of tangible systems. The second system presented in this dissertation is a tangible tabletop scheduling application. Two studies with single and paired users provide several insights into the impact of epistemic actions on the user experience when these are performed outside of a system’s sensing boundaries. These insights are clustered by the form, size and location of ideal interface areas for such offline epistemic actions to occur, as well as how can physical tokens be designed to better support them. Finally, and based on the results obtained to this point, the last study presented in this dissertation directly addresses the lack of empirical tools to formally evaluate tangible interaction. It presents a video-coding framework grounded on a systematic literature review of 78 papers, and evaluates its value as metric through a 60 participant study performed across three different research laboratories. The results highlight the usefulness and power of epistemic actions as a performance metric for tangible systems. In sum, through the use of such novel metrics in each of the three studies presented, this dissertation provides a better understanding of the real impact and benefits of designing and developing systems that feature tangible interaction.

Motivating sustainable goal choices and providing effective feedback

Overconsumption of natural resources and the associated environmental hazards are one of today’s most pressing global issues. In the western world, individual consumption in homes and workplaces is a key contributor to this problem. Reflecting the importance of individual action in this domain, this thesis focuses on studying and influencing choices related to sustainability and energy consumption made by people in their daily lives. There are three main components to this work. Firstly, this thesis asserts that people frequently make ineffective consumption reduction goal choices and attempts to understand the rationale for these poor choices by fitting them to goalsetting theory, an established theoretical model of behavior change. Secondly, it presents two approaches that attempt to influence goal choice towards more effective targets, one of which deals with mechanisms for goal priming and the other of which explores the idea that carefully designed toys can exert influence on children’s long term consumption behavior patterns. The final section of this thesis deals with the design of feedback to support the performance of environmentally sound activities. Key contributions surrounding goals include the finding that people choose easy sustainable goals despite immediate feedback as to their ineffectiveness and the discussion and study of goal priming mechanisms that can influence this choice process. Contributions within the design of value instilling toys include a theoretically grounded framework for the design of such toys and a completed and tested prototype toy. Finally, contributions in designing effective and engaging energy consumption feedback include the finding that negative feedback is best presented verbally compared with visually and this is exemplified and presented within a working feedback system. The discussions, concepts, prototypes and empirical findings presented in this work will be useful for both environmental psychologists and for HCI researchers studying eco-feedback.

Exploring the dimensions of eco-feedback in the wild

Worldwide reports about energy usage have noted the importance of the domestic energy consumption sector in the worldwide scenario. This fact motivated and supported the birth and spread of the so-called eco-feedback devices. Such devices provide information about individual or group energy consumption behavior with the goal of reducing the impact in the environment. Motivated by the body of work which questions the long term effectiveness of eco-feedback systems, this thesis focus on evaluating in-the-wild the long term usage of eco-feedback systems. We have conducted five long term studies with different eco-feedback systems designed to evaluate different dimensions in the design of eco-feedback plus two more focused short term studies aimed at studying concrete approaches. Our summary reports on the fact that the there is a novelty effect associated with ecofeedback systems in which the usage of these devices declines significantly after a month. We did not found evidence that the novelty effect is related to location or the type of information represented, nor that the decrease in the eco-feedback usage could lead to the consumption relapsing to values previous to the introduction of the eco-feedback. Our work has also generated other contributions related to the positioning of the feedback, using metaphors for representing the consumption and presenting information about the source of the energy in the feedback.

Interactive music visualization: implementation, realization and evaluation

This thesis describes all process of the development of music visualization, starting with the implementation, followed by realization and then evaluation. The main goal is to have to knowledge of how the audience live performance experience can be enhanced through music visualization. With music visualization is possible to give a better understanding about the music feelings constructing an intensive atmosphere in the live music performance, which enhances the connection between the live music and the audience through visuals. These visuals have to be related to the live music, furthermore has to quickly respond to live music changes and introduce novelty into the visuals. The mapping between music and visuals is the focus of this project, in order to improve the relationship between the live performance and the spectators. The implementation of music visualization is based on the translation of music into graphic visualizations, therefore at the beginning the project was based on the existent works. Later on, it was decided to introduce new ways of conveying music into visuals. Several attempts were made in order to discover the most efficient mapping between music and visualization so people can fully connect with the performance. Throughout this project, those attempts resulted in several music visualizations created for four live music performances, afterwards it was produced an online survey to evaluate those live performances with music visualization. In the end, all conclusions are presented based on the results of the online survey, and also is explained which music elements should be depicted in the visuals, plus how those visuals should respond to the selected music elements.