TIMPANO: Technology for complex Human-Machine conversational interaction with dynamic learning

El proyecto TIMPANO tiene por objetivo profundizar en el desarrollo de sistemas de comunicación oral hombre-máquina atendiendo principalmente a la capacidad de dar respuesta a múltiples requerimientos de los usuarios, como pueden ser el acceso a información, la extracción de información, o el análisis de grandes repositorios de información en audio. En el proyecto se hace especial énfasis en la adaptación dinámica de los modelos a diversos contextos, tanto de tipo acústico, como semántico o de idioma.

Multimodal 'Eyes-Free' interaction techniques for mobile devices

Mobile and wearable computers present input/output prob-lems due to limited screen space and interaction techniques. When mobile, users typically focus their visual attention on navigating their environment - making visually demanding interface designs hard to operate. This paper presents two multimodal interaction techniques designed to overcome these problems and allow truly mobile, 'eyes-free' device use. The first is a 3D audio radial pie menu that uses head gestures for selecting items. An evaluation of a range of different audio designs showed that egocentric sounds re-duced task completion time, perceived annoyance, and al-lowed users to walk closer to their preferred walking speed. The second is a sonically enhanced 2D gesture recognition system for use on a belt-mounted PDA. An evaluation of the system with and without audio feedback showed users' ges-tures were more accurate when dynamically guided by au-dio-feedback. These novel interaction techniques demon-strate effective alternatives to visual-centric interface de-signs on mobile devices.

Guidelines for audio-enhancement of graphical user Interface widgets

Audio feedback remains little used in most graphical user interfaces despite its potential to greatly enhance interaction. Not only does sonic enhancement of interfaces permit more natural human-computer communication but it also allows users to employ an appropriate sense to solve a problem rather than having to rely solely on vision. Research shows that designers do not typically know how to use sound effectively; subsequently, their ad hoc use of sound often leads to audio feedback being considered an annoying distraction. Unlike the design of purely graphical user interfaces for which guidelines are common, the audio-enhancement of graphical user interfaces has (until now) been plagued by a lack of suitable guidance. This paper presents a series of empirically substantiated guidelines for the design and use of audio-enhanced graphical user interface widgets.

Microteaching Lesson Study: Mentor interaction structure and its relation to elementary preservice mathematics teacher knowledge development

This study investigated Microteaching Lesson Study (MLS) and three possible MLS mentor interaction structures during the debriefing sessions in relation to elementary preservice teacher development of knowledge for teaching. One hundred three elementary preservice teachers enrolled in five different sections of a mathematics methods course at a southern urban university were part of the study. This included 72 participants who completed MLS across three different mentor interaction structures as part of their course requirements and 31 elementary preservice teachers who did not complete MLS as part of their methods course and served as a comparison group for a portion of the study. A sequential mixed-methods research design was used to analyze the relationship between MLS mentor interaction structure and growth in preservice teachers' mathematics teacher knowledge. Data sources included pre and post assessments, group developed lesson plans and final reports, a feedback survey with Likert-type and open-ended questions, and transcripts of audio-recorded debriefing sessions. The pre and post assessments were analyzed using Analysis of Variance (ANOVA) and descriptive statistics were used to analyze the Likert-type feedback survey questions. Group MLS lesson plans, final reports, and transcripts of debriefing sessions along with the open-ended questions from the feedback survey were coded in a three-step process as described by Miles and Huberman (1994). In alignment with findings from M. Fernandez (2005, 2010), elementary preservice teachers participating in MLS grew in content knowledge related to MLS topics taught by one another. Results from the analysis of pre and post content knowledge assessments revealed that participants grew in their understanding of the mathematics topics taught during MLS irrespective of their mentor interaction structure and when compared to the participants who did not complete MLS in their methods course. Findings from the analysis of lesson plans for growth in pedagogical content knowledge revealed the most growth in this area occurred for participants assigned to the interaction structure in which the MLS mentor participated in the first two debriefing sessions. Analysis of the transcripts of the discourse during the debriefing sessions and the feedback surveys support the finding that the elementary preservice teachers assigned to the interaction structure in which the MLS mentor participated in the first and second debriefing sessions benefited more from the MLS experience when compared to elementary preservice teachers assigned to the other two interaction structures (MLS mentor participated in only the first debriefing session and MLS mentor participated in only the last debriefing session).

'Scratch' locative audio drama - report to BBC executives on trial at BBC FreeThinking 2008

‘Scratch’ investigates the use of physical space as a representation of narrative and dramatic structure. An audio-drama, it is a world-first in being location-sensitive without being tied to any particular place (preceding attempts by others have emphasised location-specific aspects of the genre). Developed in collaboration with and part-funded by BBC Radio Drama, it builds on research undertaken for ‘Dragons’ (output 4). It uses pre-recorded audio on GPS-enabled mobile devices allowing sounds to be virtually attached to locations in an outdoor space. As participants move, they encounter scenes forming a coherent drama which behave differently if the same place is visited more than once. This translocational approach opens novel artistic possibilities exploited through team expertise in narrative, sound design and advanced interaction. It is also significant in the economics of broadcast media as a more viable proposition than the many experimental locative experiences which have been site-specific: this was of great interest to the BBC. The public performance selected for BBC FreeThinking, 1-2 September 2008 in Liverpool as part of European Capital of Culture was reported in a co-authored 2009 conference presentation at ISEA, Belfast, 26-29 August 2009 and in a co-authored short chapter in Spierling and Szilas (eds.) Interactive Storytelling, Springer 2008. Boyd Davis directed the project and devised and undertook the evaluation with 40 trial listeners, reporting to BBC executives (http://researchonline.rca.ac.uk/1000/) for whom a second trial was also run in London in 2009. The evaluation used interview, video observation and a questionnaire combining an open question at the beginning with more specific questions later, avoiding channelling respondents' reactions immediately after the experience into issues which might not be uppermost in their minds, while also yielding data capable of rigorous analysis. The evaluation was to provide feedback to the makers of the drama and to guide policy at the BBC. [287] Participants were recruited principally through the publicity for FreeThinking 2008 – mainly via the festival website. The average age of participants was 40. The gender of participants was 20 males, 17 females and 3 null returns. The evaluation strategy was to combine an open question at the beginning with more specific questions later. In this way we avoided channeling respondents' initial opinions immediately after the experience into issues which might not be uppermost in their minds, while also yielding data capable of rigorous analysis. The purpose of the evaluation was to provide guidance for ourselves as the makers of the drama and to guide policy at the BBC on locative and other interactive media. The responses are analysed in the report.

Playing with sound and gesture in digital audio games

This article introduces the genre of a digital audio game and discusses selected play interaction solutions implemented in the Audio Game Hub, a prototype designed and evaluated in the years 2014 and 2015 at the Gamification Lab at Leuphana University Lüneburg.1 The Audio Game Hub constitutes a set of familiar playful activities (aiming at a target, reflex-based reacting to sound signals, labyrinth exploration) and casual games (e.g. Tetris, Memory) adapted to the digital medium and converted into the audio sphere, where the player is guided predominantly or solely by sound. The authors will discuss the design questions raised at early stages of the project, and confront them with the results of user experience testing performed on two groups of sighted and one group of visually impaired gamers.

Developmental Neural Correlates of Social Interaction

Children develop in a sea of reciprocal social interaction, but their brain development is predominately studied in non-interactive contexts (e.g., viewing photographs of faces). This dissertation investigated how the developing brain supports social interaction. Specifically, novel paradigms were used to target two facets of social experience—social communication and social motivation—across three studies in children and adults. In Study 1, adults listened to short vignettes—which contained no social information—that they believed to be either prerecorded or presented over an audio-feed by a live social partner. Simply believing that speech was from a live social partner increased activation in the brain’s mentalizing network—a network involved in thinking about others’ thoughts. Study 2 extended this paradigm to middle childhood, a time of increasing social competence and social network complexity, as well as structural and functional social brain development. Results showed that, as in adults, regions of the mentalizing network were engaged by live speech. Taken together, these findings indicate that the mentalizing network may support the processing of interactive communicative cues across development. Given this established importance of social-interactive context, Study 3 examined children’s social motivation when they believed they were engaged in a computer-based chat with a peer. Children initiated interaction via sharing information about their likes and hobbies and received responses from the peer. Compared to a non-social control, in which children chatted with a computer, peer interaction increased activation in mentalizing regions and reward circuitry. Further, within mentalizing regions, responsivity to the peer increased with age. Thus, across all three studies, social cognitive regions associated with mentalizing supported real-time social interaction. In contrast, the specific social context appeared to influence both reward circuitry involvement and age-related changes in neural activity. Future studies should continue to examine how the brain supports interaction across varied real-world social contexts. In addition to illuminating typical development, understanding the neural bases of interaction will offer insight into social disabilities such as autism, where social difficulties are often most acute in interactive situations. Ultimately, to best capture human experience, social neuroscience ought to be embedded in the social world.

Interaction techniques with novel multimodal feedback for addressing gesture-sensing systems

Users need to be able to address in-air gesture systems, which means finding where to perform gestures and how to direct them towards the intended system. This is necessary for input to be sensed correctly and without unintentionally affecting other systems. This thesis investigates novel interaction techniques which allow users to address gesture systems properly, helping them find where and how to gesture. It also investigates audio, tactile and interactive light displays for multimodal gesture feedback; these can be used by gesture systems with limited output capabilities (like mobile phones and small household controls), allowing the interaction techniques to be used by a variety of device types. It investigates tactile and interactive light displays in greater detail, as these are not as well understood as audio displays. Experiments 1 and 2 explored tactile feedback for gesture systems, comparing an ultrasound haptic display to wearable tactile displays at different body locations and investigating feedback designs. These experiments found that tactile feedback improves the user experience of gesturing by reassuring users that their movements are being sensed. Experiment 3 investigated interactive light displays for gesture systems, finding this novel display type effective for giving feedback and presenting information. It also found that interactive light feedback is enhanced by audio and tactile feedback. These feedback modalities were then used alongside audio feedback in two interaction techniques for addressing gesture systems: sensor strength feedback and rhythmic gestures. Sensor strength feedback is multimodal feedback that tells users how well they can be sensed, encouraging them to find where to gesture through active exploration. Experiment 4 found that they can do this with 51mm accuracy, with combinations of audio and interactive light feedback leading to the best performance. Rhythmic gestures are continuously repeated gesture movements which can be used to direct input. Experiment 5 investigated the usability of this technique, finding that users can match rhythmic gestures well and with ease. Finally, these interaction techniques were combined, resulting in a new single interaction for addressing gesture systems. Using this interaction, users could direct their input with rhythmic gestures while using the sensor strength feedback to find a good location for addressing the system. Experiment 6 studied the effectiveness and usability of this technique, as well as the design space for combining the two types of feedback. It found that this interaction was successful, with users matching 99.9% of rhythmic gestures, with 80mm accuracy from target points. The findings show that gesture systems could successfully use this interaction technique to allow users to address them. Novel design recommendations for using rhythmic gestures and sensor strength feedback were created, informed by the experiment findings.

Peer interaction in mixed age groups: a study in the computer area of an early childhood education center in Portugal

The study was developed as a teacher-research project during initial teacher education – Masters Degree of Early Childhood and Primary Education, in Portugal. It analysed the interactions between children of 3 to 6 years old, during the use of the computer as a free choice activity, confronting situations between peers of the same age and situations between peers of different ages. The focus of the analysis was the collaborative interactions. This was a qualitative study. Children could choose the computer, amongst other interest areas, and work for around an hour in pairs. In the computer, children used mainly educational games. During four weeks, the interactions between the pairs were audio recorded. Field notes and informal interviews to the children were also used to collect data. Eleven children were involved in the study with ages ranging from 3 to 6 years old. Baseline data on children’s basic computer proficiency was collected using the Individualized Computer Proficiency Checklist (ICPC) by Hyun. The recorded interactions were analysed using the types of talk offered by Scrimshaw and Perkins and Wegerif and Scrimshaw: cumulative talk, exploratory talk, disputational talk, and tutorial talk. This framework was already used in a study in an early childhood education context in Portugal by Amante. The results reveal differences in computer use and characterize the observed interactions. Seven different pairs of children's interactions were analysed. More than a third of the interactions were cumulative talk, followed by exploratory talk, tutorial talk and disputational talk. Comparing same and mixed age pairs, we observed that cumulative talk is the more present interaction, but in same age pairs this is followed by exploratory talk whereas in the mixed age pairs it is tutorial talk that has the second largest percentage. The pairs formed by the children were very asymmetrical in terms of age and computer proficiency. This lead to the more tutorial interactions, where one children showed the other or directed him/her on how to play. The results show that collaboration is present during the use of a computer area in early childhood education. The free choice of the children means the adults can only suggest pairing suited to specific interactions between the children. Another way to support children in more exploratory talk interactions could be by discussing the way the older children can help the younger ones beyond directing or correcting their work.