Quantifying epistemic actions in human-computer interaction

As digital systems move away from traditional desktop setups, new interaction paradigms are emerging that better integrate with users’ realworld surroundings, and better support users’ individual needs. While promising, these modern interaction paradigms also present new challenges, such as a lack of paradigm-specific tools to systematically evaluate and fully understand their use. This dissertation tackles this issue by framing empirical studies of three novel digital systems in embodied cognition – an exciting new perspective in cognitive science where the body and its interactions with the physical world take a central role in human cognition. This is achieved by first, focusing the design of all these systems on a contemporary interaction paradigm that emphasizes physical interaction on tangible interaction, a contemporary interaction paradigm; and second, by comprehensively studying user performance in these systems through a set of novel performance metrics grounded on epistemic actions, a relatively well established and studied construct in the literature on embodied cognition. The first system presented in this dissertation is an augmented Four-in-a-row board game. Three different versions of the game were developed, based on three different interaction paradigms (tangible, touch and mouse), and a repeated measures study involving 36 participants measured the occurrence of three simple epistemic actions across these three interfaces. The results highlight the relevance of epistemic actions in such a task and suggest that the different interaction paradigms afford instantiation of these actions in different ways. Additionally, the tangible version of the system supports the most rapid execution of these actions, providing novel quantitative insights into the real benefits of tangible systems. The second system presented in this dissertation is a tangible tabletop scheduling application. Two studies with single and paired users provide several insights into the impact of epistemic actions on the user experience when these are performed outside of a system’s sensing boundaries. These insights are clustered by the form, size and location of ideal interface areas for such offline epistemic actions to occur, as well as how can physical tokens be designed to better support them. Finally, and based on the results obtained to this point, the last study presented in this dissertation directly addresses the lack of empirical tools to formally evaluate tangible interaction. It presents a video-coding framework grounded on a systematic literature review of 78 papers, and evaluates its value as metric through a 60 participant study performed across three different research laboratories. The results highlight the usefulness and power of epistemic actions as a performance metric for tangible systems. In sum, through the use of such novel metrics in each of the three studies presented, this dissertation provides a better understanding of the real impact and benefits of designing and developing systems that feature tangible interaction.

Motivating sustainable goal choices and providing effective feedback

Overconsumption of natural resources and the associated environmental hazards are one of today’s most pressing global issues. In the western world, individual consumption in homes and workplaces is a key contributor to this problem. Reflecting the importance of individual action in this domain, this thesis focuses on studying and influencing choices related to sustainability and energy consumption made by people in their daily lives. There are three main components to this work. Firstly, this thesis asserts that people frequently make ineffective consumption reduction goal choices and attempts to understand the rationale for these poor choices by fitting them to goalsetting theory, an established theoretical model of behavior change. Secondly, it presents two approaches that attempt to influence goal choice towards more effective targets, one of which deals with mechanisms for goal priming and the other of which explores the idea that carefully designed toys can exert influence on children’s long term consumption behavior patterns. The final section of this thesis deals with the design of feedback to support the performance of environmentally sound activities. Key contributions surrounding goals include the finding that people choose easy sustainable goals despite immediate feedback as to their ineffectiveness and the discussion and study of goal priming mechanisms that can influence this choice process. Contributions within the design of value instilling toys include a theoretically grounded framework for the design of such toys and a completed and tested prototype toy. Finally, contributions in designing effective and engaging energy consumption feedback include the finding that negative feedback is best presented verbally compared with visually and this is exemplified and presented within a working feedback system. The discussions, concepts, prototypes and empirical findings presented in this work will be useful for both environmental psychologists and for HCI researchers studying eco-feedback.

Exploring the dimensions of eco-feedback in the wild

Worldwide reports about energy usage have noted the importance of the domestic energy consumption sector in the worldwide scenario. This fact motivated and supported the birth and spread of the so-called eco-feedback devices. Such devices provide information about individual or group energy consumption behavior with the goal of reducing the impact in the environment. Motivated by the body of work which questions the long term effectiveness of eco-feedback systems, this thesis focus on evaluating in-the-wild the long term usage of eco-feedback systems. We have conducted five long term studies with different eco-feedback systems designed to evaluate different dimensions in the design of eco-feedback plus two more focused short term studies aimed at studying concrete approaches. Our summary reports on the fact that the there is a novelty effect associated with ecofeedback systems in which the usage of these devices declines significantly after a month. We did not found evidence that the novelty effect is related to location or the type of information represented, nor that the decrease in the eco-feedback usage could lead to the consumption relapsing to values previous to the introduction of the eco-feedback. Our work has also generated other contributions related to the positioning of the feedback, using metaphors for representing the consumption and presenting information about the source of the energy in the feedback.

BlueFriends: measuring, analyzing and preventing social exclusion between elementary school students

Social exclusion is a relatively recent term, whose creation is attributed to René Lenoir(Lenoir, 1974). Its concept covers a remarkably wide range of social and economic problems, and can be triggered for various reasons: mentally and physically handicapped, abused children, delinquents, multi-problem households, asocial people, and other social “misfits” (Silver, 1995, pp. 63; Foucault, 1992). With an increasingly multi-cultural population, cultural and social inequalities rapidly ascend, bringing with them the need for educational restructuring. We are living in an evermore diverse world, and children need to be educated to be receptive to the different types of people around them, especially considering social and cultural aspects. It is with these goals that inclusive education has seen an increased trend in today’s academic environment, reminding us that even though children may be taught under the same roof, discriminatory practices might still happen. There are, however, a number of developed tools to assess the various dimensions of social networks. These are mostly based on questionnaires and interviews, which tend to be fastidious and don’t allow for longitudinal, large scale measurement. This thesis introduces BlueFriends, a Bluetooth-based measurement tool for social inclusion/exclusion on elementary school classes. The main goals behind the development of this tool were a) understanding how exclusion manifests in students’ behaviors, and b) motivating pro-social behaviors on children through the use of a persuasive technology. BlueFriends is a distributed application, comprised by an application running on several smartphones, a web-hosted database and a computer providing a visual representation of the data collected on a TV screen, attempting to influence children behaviors. The application makes use of the Bluetooth device present on each phone to continuously sample the RSSI (Received Signal Strength Indication) from other phones, storing the data locally on each phone. All of the stored data is collected, processed and then inserted into the database at the end of each day. At the beginning of each recess, children are reminded of how their behaviors affect others with the help of a visual display, which consists of interactions between dogs. This display illustrates every child’s best friends, as well as which colleagues they don’t interact with as much. Several tips encouraging social interaction and inclusiveness are displayed, inspiring children to change their behaviors towards the colleagues they spend less time with. This thesis documents the process of designing, deploying and analyzing the results of two field studies. On the first study, we assess how the current developed tools are inferior to our measuring tool by deploying a measurement only study, aimed at perceiving how much information can be obtained by the BlueFriends application and attempting to understand how exclusion manifests itself in the school environment. On the second study, we pile on the previous to try and motivate pro-social behaviors on students, with the use of visual cues and recommendations. Ultimately, we confirm that our measurement tool’s results were satisfying towards measuring and changing children’s behaviors, and conclude with our thoughts on possible future work, suggesting a number of possible extensions and improvements.

3D reconstruction through photographs

Humans can perceive three dimension, our world is three dimensional and it is becoming increasingly digital too. We have the need to capture and preserve our existence in digital means perhaps due to our own mortality. We have also the need to reproduce objects or create small identical objects to prototype, test or study them. Some objects have been lost through time and are only accessible through old photographs. With robust model generation from photographs we can use one of the biggest human data sets and reproduce real world objects digitally and physically with printers. What is the current state of development in three dimensional reconstruction through photographs both in the commercial world and in the open source world? And what tools are available for a developer to build his own reconstruction software? To answer these questions several pieces of software were tested, from full commercial software packages to open source small projects, including libraries aimed at computer vision. To bring to the real world the 3D models a 3D printer was built, tested and analyzed, its problems and weaknesses evaluated. Lastly using a computer vision library a small software with limited capabilities was developed.

A multimodal tablet-based interface for designing and reviewing 3D engineering models

The ability to view and interact with 3D models has been happening for a long time. However, vision-based 3D modeling has only seen limited success in applications, as it faces many technical challenges. Hand-held mobile devices have changed the way we interact with virtual reality environments. Their high mobility and technical features, such as inertial sensors, cameras and fast processors, are especially attractive for advancing the state of the art in virtual reality systems. Also, their ubiquity and fast Internet connection open a path to distributed and collaborative development. However, such path has not been fully explored in many domains. VR systems for real world engineering contexts are still difficult to use, especially when geographically dispersed engineering teams need to collaboratively visualize and review 3D CAD models. Another challenge is the ability to rendering these environments at the required interactive rates and with high fidelity. In this document it is presented a virtual reality system mobile for visualization, navigation and reviewing large scale 3D CAD models, held under the CEDAR (Collaborative Engineering Design and Review) project. It’s focused on interaction using different navigation modes. The system uses the mobile device's inertial sensors and camera to allow users to navigate through large scale models. IT professionals, architects, civil engineers and oil industry experts were involved in a qualitative assessment of the CEDAR system, in the form of direct user interaction with the prototypes and audio-recorded interviews about the prototypes. The lessons learned are valuable and are presented on this document. Subsequently it was prepared a quantitative study on the different navigation modes to analyze the best mode to use it in a given situation.

Interactive music visualization: implementation, realization and evaluation

This thesis describes all process of the development of music visualization, starting with the implementation, followed by realization and then evaluation. The main goal is to have to knowledge of how the audience live performance experience can be enhanced through music visualization. With music visualization is possible to give a better understanding about the music feelings constructing an intensive atmosphere in the live music performance, which enhances the connection between the live music and the audience through visuals. These visuals have to be related to the live music, furthermore has to quickly respond to live music changes and introduce novelty into the visuals. The mapping between music and visuals is the focus of this project, in order to improve the relationship between the live performance and the spectators. The implementation of music visualization is based on the translation of music into graphic visualizations, therefore at the beginning the project was based on the existent works. Later on, it was decided to introduce new ways of conveying music into visuals. Several attempts were made in order to discover the most efficient mapping between music and visualization so people can fully connect with the performance. Throughout this project, those attempts resulted in several music visualizations created for four live music performances, afterwards it was produced an online survey to evaluate those live performances with music visualization. In the end, all conclusions are presented based on the results of the online survey, and also is explained which music elements should be depicted in the visuals, plus how those visuals should respond to the selected music elements.

DEMO models based automatic workflow process generation

Nowadays, more than half of the computer development projects fail to meet the final users' expectations. One of the main causes is insufficient knowledge about the organization of the enterprise to be supported by the respective information system. The DEMO methodology (Design and Engineering Methodology for Organizations) has been proved as a well-defined method to specify, through models and diagrams, the essence of any organization at a high level of abstraction. However, this methodology is platform implementation independent, lacking the possibility of saving and propagating possible changes from the organization models to the implemented software, in a runtime environment. The Universal Enterprise Adaptive Object Model (UEAOM) is a conceptual schema being used as a basis for a wiki system, to allow the modeling of any organization, independent of its implementation, as well as the previously mentioned change propagation in a runtime environment. Based on DEMO and UEAOM, this project aims to develop efficient and standardized methods, to enable an automatic conversion of DEMO Ontological Models, based on UEAOM specification into BPMN (Business Process Model and Notation) models of processes, using clear semantics, without ambiguities, in order to facilitate the creation of processes, almost ready for being executed on workflow systems that support BPMN.

Organizations redesign and building of information systems

Develop software is still a risky business. After 60 years of experience, this community is still not able to consistently build Information Systems (IS) for organizations with predictable quality, within previously agreed budget and time constraints. Although software is changeable we are still unable to cope with the amount and complexity of change that organizations demand for their IS. To improve results, developers followed two alternatives: Frameworks that increase productivity but constrain the flexibility of possible solutions; Agile ways of developing software that keep flexibility with less upfront commitments. With strict frameworks, specific hacks have to be put in place to get around the framework construction options. In time this leads to inconsistent architectures that are harder to maintain due to incomplete documentation and human resources turnover. The main goals of this work is to create a new way to develop flexible IS for organizations, using web technologies, in a faster, better and cheaper way that is more suited to handle organizational change. To do so we propose an adaptive object model that uses a new ontology for data and action with strict normalizing rules. These rules should bound the effects of changes that can be better tested and therefore corrected. Interfaces are built with templates of resources that can be reused and extended in a flexible way. The “state of the world” for each IS is determined by all production and coordination acts that agents performed over time, even those performed by external systems. When bugs are found during maintenance, their past cascading effects can be checked through simulation, re-running the log of transaction acts over time and checking results with previous records. This work implements a prototype with part of the proposed system in order to have a preliminary assessment its feasibility and limitations.

Hospital hero: a game for reducing stress and anxiety of children while waiting in emergency room

This report tells a story which started as an idea that came to us to fight the battle-cry feeling commonly known as stress and anxiety. Before creating the solution of the idea, we first need to understand the feelings underneath and its effects on our well-being. Throughout the course of our lives, we experience states of weakness and fear. These feelings can arise, for instance, while we are in an emergency room. Needless to say, how much it would have imaginable effects on children, who are unfamiliar to such environments. We ran through a serious of scenarios to find the most suitable solution, among them the study of interaction with positive expressions by Dr. Baldwin, proved to be a valued resource. It was reduced due to its length and to be suitable to our public audience. The game was then created in order to reduce or even eliminate the stress and anxiety of children. Since the game was initially released, some modifications had been made but the original idea - interaction with positive expressions – remained. When the time came, we asked children to play one of the two versions of the game while waiting in the emergency room. This not only created a diversion for them but also a learning experience as it displayed some hospital equipment. The difference between the two versions is that one provides expressions, while the other does not. After all our hard work, we felt rewarded because the project proved its worth and we would see that in the expressions on children’s faces while they played. Most importantly, their anxiety level numbers were significantly reduced during that short period of time.

Color measurement using a smartphone applied to Madeira wines

Computer vision is a field that uses techniques to acquire, process, analyze and understand images from the real world in order to produce numeric or symbolic information in the form of decisions [1]. This project aims to use computer vision to prepare an app to analyze a Madeira Wine and characterize it (identify its variety) by its color. Dry or sweet wines, young or old wines have a specific color. It uses techniques to compare histograms in order to analyze the images taken from a test sample inside a special container designed for this purpose. The color analysis from a wine sample using an image captured by a smartphone can be difficult. Many factors affect the captured image such as, light conditions, the background of the sample container due to the many positions the photo can be taken (different to capture facing a white wall or facing the floor for example). Using new technologies such as 3D printing it was possible to create a prototype that aims to control the effect of those external factors on the captured image. The results for this experiment are good indicators for future works. Although it’s necessary to do more tests, the first tests had a success rate of 80% to 90% of correct results. This report documents the development of this project and all the techniques and steps required to execute the tests.

Integrating eye tracking in virtual reality for stroke rehabilitation

This thesis reports on research done for the integration of eye tracking technology into virtual reality environments, with the goal of using it in rehabilitation of patients who suffered from stroke. For the last few years, eye tracking has been a focus on medical research, used as an assistive tool  to help people with disabilities interact with new technologies  and as an assessment tool  to track the eye gaze during computer interactions. However, tracking more complex gaze behaviors and relating them to motor deficits in people with disabilities is an area that has not been fully explored, therefore it became the focal point of this research. During the research, two exploratory studies were performed in which eye tracking technology was integrated in the context of a newly created virtual reality task to assess the impact of stroke. Using an eye tracking device and a custom virtual task, the system developed is able to monitor the eye gaze pattern changes over time in patients with stroke, as well as allowing their eye gaze to function as an input for the task. Based on neuroscientific hypotheses of upper limb motor control, the studies aimed at verifying the differences in gaze patterns during the observation and execution of the virtual goal-oriented task in stroke patients (N=10), and also to assess normal gaze behavior in healthy participants (N=20). Results were found consistent and supported the hypotheses formulated, showing that eye gaze could be used as a valid assessment tool on these patients. However, the findings of this first exploratory approach are limited in order to fully understand the effect of stroke on eye gaze behavior. Therefore, a novel model-driven paradigm is proposed to further understand the relation between the neuronal mechanisms underlying goal-oriented actions and eye gaze behavior.

SysPRE: systematized process for requirements engineering in knowledge discovery projects

The domain of Knowledge Discovery (KD) and Data Mining (DM) is of growing importance in a time where more and more data is produced and knowledge is one of the most precious assets. Having explored both the existing underlying theory, the results of the ongoing research in academia and the industry practices in the domain of KD and DM, we have found that this is a domain that still lacks some systematization. We also found that this systematization exists to a greater degree in the Software Engineering and Requirements Engineering domains, probably due to being more mature areas. We believe that it is possible to improve and facilitate the participation of enterprise stakeholders in the requirements engineering for KD projects by systematizing requirements engineering process for such projects. This will, in turn, result in more projects that end successfully, that is, with satisfied stakeholders, including in terms of time and budget constraints. With this in mind and based on all information found in the state-of-the art, we propose SysPRE - Systematized Process for Requirements Engineering in KD projects. We begin by proposing an encompassing generic description of the KD process, where the main focus is on the Requirements Engineering activities. This description is then used as a base for the application of the Design and Engineering Methodology for Organizations (DEMO) so that we can specify a formal ontology for this process. The resulting SysPRE ontology can serve as a base that can be used not only to make enterprises become aware of their own KD process and requirements engineering process in the KD projects, but also to improve such processes in reality, namely in terms of success rate.

De touch: extending service channels in an airport through touch technologies

This project aimed to create a communication and interaction channel between Madeira Airport and its passengers. We used the pre-existent touch enabled screens at the terminal since their potential was not being utilised to their full capacity. To achieve our goal, we have followed an agile strategy to create a testable prototype and take advantages of its results. The developed prototype is based on a plugin architecture turning it into a maintainable and highly customisable system. The collected usage data suggests that we have achieved the initially defined goals. There is no doubt that this new interaction channel is an improvement regarding the provided services and, supported by the usage data, there is an opportunity to explore additional developments to the channel.

OpenDataHub: an open dataset management system

This thesis presents a cloud-based software platform for sharing publicly available scientific datasets. The proposed platform leverages the potential of NoSQL databases and asynchronous IO technologies, such as Node.JS, in order to achieve high performances and flexible solutions. This solution will serve two main groups of users. The dataset providers, which are the researchers responsible for sharing and maintaining datasets, and the dataset users, that are those who desire to access the public data. To the former are given tools to easily publish and maintain large volumes of data, whereas the later are given tools to enable the preview and creation of subsets of the original data through the introduction of filter and aggregation operations. The choice of NoSQL over more traditional RDDMS emerged from and extended benchmark between relational databases (MySQL) and NoSQL (MongoDB) that is also presented in this thesis. The obtained results come to confirm the theoretical guarantees that NoSQL databases are more suitable for the kind of data that our system users will be handling, i. e., non-homogeneous data structures that can grow really fast. It is envisioned that a platform like this can lead the way to a new era of scientific data sharing where researchers are able to easily share and access all kinds of datasets, and even in more advanced scenarios be presented with recommended datasets and already existing research results on top of those recommendations.