Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses

Story understanding involves many perceptual and cognitive subprocesses, from perceiving individual words, to parsing sentences, to understanding the relationships among the story characters. We present an integrated computational model of reading that incorporates these and additional subprocesses, simultaneously discovering their fMRI signatures. Our model predicts the fMRI activity associated with reading arbitrary text passages, well enough to distinguish which of two story segments is being read with 74% accuracy. This approach is the first to simultaneously track diverse reading subprocesses during complex story processing and predict the detailed neural representation of diverse story features, ranging from visual word properties to the mention of different story characters and different actions they perform. We construct brain representation maps that replicate many results from a wide range of classical studies that focus each on one aspect of language processing and offer new insights on which type of information is processed by different areas involved in language processing. Additionally, this approach is promising for studying individual differences: it can be used to create single subject maps that may potentially be used to measure reading comprehension and diagnose reading disorders.

The anticipatory space of the bunker, modernity’s dark mirror

The buried and semi-buried bunker, bulwark since the early eighteenth century against increasingly sophisticated forms of ordnance, emerged in increasing number in Europe throughout the twentieth century across a series of scales from the household Anderson shelter to the vast infrastructural works of the Maginot and Siegfried lines, or the Atlantic Wall. Its latest proliferation took place during the Cold War. From these perspectives, it is as emblematic of modernity as the department store, the great exhibition, the skyscraper or the machine-inspired domestic space advocated by Le Corbusier. It also represents the obverse, or perhaps a parodic iteration, of the preoccupations of early architectural modernism: a vast underground international style, cast in millions of tons of thick, reinforced concrete retaining walls, whose spatial relationship to the landscape above was strictly mediated through the periscope, the loop-hole, the range finder and the strategic necessity to both resist and facilitate the technologies and scopic regimes of weaponry. Embarking from Bunker Archaeology, this paper critically uncoils Paul Virillo’s observation, that once physically eclipsed in its topographical and technical settings, the bunker’s efficacy would mutate to other domains, retaining and remaking its meaning in another topology during the Cold War. ‘The essence of the new fortress’ he writes ‘is elsewhere, underfoot, invisible from here on in’. Shaped by this impulse, this paper seeks to render visible the bunker’s significance in a wider milieu and, in doing so, excavate some of the relationships between the physical artefact, its implications and its enduring metaphorical and perceptual ghosts.

Sonification as Concurrent Augmented Feedback for Motor Skill Learning and the Importance of Mapping Design

In recent years, sonification of movement has emerged as a viable method for the provision of feedback in motor learning. Despite some experimental validation of its utility, controlled trials to test the usefulness of sonification in a motor learning context are still rare. As such, there are no accepted conventions for dealing with its implementation. This article addresses the question of how continuous movement information should be best presented as sound to be fed back to the learner. It is proposed that to establish effective approaches to using sonification in this context, consideration must be given to the processes that underlie motor learning, in particular the nature of the perceptual information available to the learner for performing the task at hand. Although sonification has much potential in movement performance enhancement, this potential is largely unrealised as of yet, in part due to the lack of a clear framework for sonification mapping: the relationship between movement and sound. By grounding mapping decisions in a firmer understanding of how perceptual information guides learning, and an embodied cognition stance in general, it is hoped that greater advances in use of sonification to enhance motor learning can be achieved.

New methods for studying perception and action coupling

To study perception and action, Gibson advocated that “the laboratory must be like life” (Gibson, 1979, p. 3). In other words, the interactive relationship between an organism and his/her envi- ronment must be maintained so that the behavior observed in an experimental context mirrors, as closely as possible, the behavior observed in a realistic sport setting. The concept of repre- sentative design introduced by Brunswik in 1956 emphasized the need to have experimental tasks that allow the player to pick up perceptual information that specifies a property of the environment-actor system (Araújo et al., 2005; see also Chapter 24). In this chapter we will provide a brief overview of the methodologies used to study perception and action in sport and present, in some detail, the opportunities new methodologies such as immersive, interactive vir- tual reality can offer researchers in sport expertise.

Of no fixed form: The Irish Pavilion at Venice 2014 and beyond

This paper explores the theme of exhibiting architectural research through a particular example, the development of the Irish pavilion for the 14th architectural biennale, Venice 2014. Responding to Rem Koolhaas’s call to investigate the international absorption of modernity, the Irish pavilion became a research project that engaged with the development of the architectures of infrastructure in Ireland in the twentieth and twenty-first centuries. Central to this proposition was that infrastructure is simultaneously a technological and cultural construct, one that for Ireland occupied a critical position in the building of a new, independent post-colonial nation state, after 1921.

Presupposing infrastructure as consisting of both visible and invisible networks, the idea of a matrix become a central conceptual and visual tool in the curatorial and design process for the exhibition and pavilion. To begin with this was a two-dimensional grid used to identify and order what became described as a series of ten ‘infrastructural episodes’. These were determined chronologically across the decades between 1914 and 2014 and their spatial manifestations articulated in terms of scale: micro, meso and macro. At this point ten academics were approached as researchers. Their purpose was twofold, to establish the broader narratives around which the infrastructures developed and to scrutinise relevant archives for compelling visual material. Defining the meso scale as that of the building, the media unearthed was further filtered and edited according to a range of categories – filmic/image, territory, building detail, and model – which sought to communicate the relationship between the pieces of architecture and the larger systems to which they connect. New drawings realised by the design team further iterated these relationships, filling in gaps in the narrative by providing composite, strategic or detailed drawings.

Conceived as an open-ended and extendable matrix, the pavilion was influenced by a series of academic writings, curatorial practices, artworks and other installations including: Frederick Kiesler’s City of Space (1925), Eduardo Persico and Marcello Nizzoli’s Medaglio d’Oro room (1934), Sol Le Witt’s Incomplete Open Cubes (1974) and Rosalind Krauss’s seminal text ‘Grids’ (1979). A modular frame whose structural bays would each hold and present an ‘episode’, the pavilion became both a visual analogue of the unseen networks embodying infrastructural systems and a reflection on the predominance of framed structures within the buildings exhibited. Sharing the aspiration of adaptability of many of these schemes, its white-painted timber components are connected by easily-dismantled steel fixings. These and its modularity allow the structure to be both taken down and re-erected subsequently in different iterations. The pavilion itself is, therefore, imagined as essentially provisional and – as with infrastructure – as having no fixed form. Presenting archives and other material over time, the transparent nature of the space allowed these to overlap visually conveying the nested nature of infrastructural production. Pursuing a means to evoke the qualities of infrastructural space while conveying a historical narrative, the exhibition’s termination in the present is designed to provoke in the visitor, a perceptual extension of the matrix to engage with the future.

Transposing Musical Skill: Sonification of movement as concurrent augmented feedback enhances learning in a bimanual task

Concurrent feedback provided during acquisition can enhance performance of novel tasks. The ‘guidance hypothesis’ predicts that feedback provision leads to dependence and poor performance in its absence. However, appropriately-structured feedback information provided through sound (‘sonification’) may not be subject to this effect. We test this directly using a rhythmic bimanual shape-tracing task in which participants learned to move at a 4:3 timing ratio. Sonification of movement and demonstration was compared to two other learning conditions: (1) sonification of task demonstration alone and (2) completely silent practice (control). Sonification of movement emerged as the most effective form of practice, reaching significantly lower error scores than control. Sonification of solely the demonstration, which was expected to benefit participants by perceptually unifying task requirements, did not lead to better performance than control. Good performance was maintained by participants in the sonification condition in an immediate retention test without feedback, indicating that the use of this feedback can overcome the guidance effect. On a 24-hour retention test, performance had declined and was equal between groups. We argue that this and similar findings in the feedback literature are best explained by an ecological approach to motor skill learning which places available perceptual information at the highest level of importance.

Voz e emoção em português europeu

No trabalho apresentado realiza-se uma primeira descrição de voz e emoção para o Português Europeu. Estudamos, utilizando como base estudos realizados em diversas línguas (finlandês; inglês; alemão), os parâmetros relacionados com voz e que variam consoante a emoção que expressamos. Analisamos assim os parâmetros relacionados com a frequência Fundamental (F0) com a perturbação (jitter) com a amplitude (shimmer) e com aspectos relacionados com o ruído (HNR). Trata-se de um estudo abrangente que estudando voz e a sua relação/variação de acordo com a emoção o faz em três vertentes: patologia de voz de origem psicogénica (carácter emocional); emoção produzida por actores e a análise de emoção espontânea. Conseguindo, como trabalho pioneiro nesta área, valores para todos estes tipos de produção. Salientamos o facto de no nosso trabalho apenas existir a análise de voz sem recurso a expressão facial ou à postura dos indivíduos. Para que pudéssemos realizar estudos comparativos com os dados que íamos recolhendo em cada corpus (patologia; emoção por actor e emoção espontânea), procurámos utilizar sempre os mesmos métodos de análise (Praat; SFS; SPSS, Hoarseness Diagram – para a análise de voz com patologia - e o sistema Feeltrace - para as emoções espontâneas). Os estudos e análises relativos à emoção produzida por actores são complementados por testes de percepção aplicados a falantes nativos de Inglês Americano e a falantes de Português Europeu. Este teste, juntamente com a análise da emoção espontânea, permitiu-nos retirar dados particulares relativos à língua portuguesa. Apesar de haver tanto na expressão como na percepção de emoções muitas características consideradas universais, em Português percebe-se algo de peculiar. Os valores para a expressão neutra; tristeza e alegria são todos muito próximos, ao contrário do que acontece noutras línguas. Além disso estas três emoções (de famílias distintas) são as que mais dificuldades causam (aos dois grupos de informantes) em termos de distinção no teste de percepção. Poderá ser esta a particularidade da expressão da emoção no Português Europeu, podendo estar ligada a factores culturais. Percebe-se ainda, com este trabalho, que a emoção expressa pelo actor se aproxima da emoção espontânea. No entanto, alguns parâmetros apresentam valores diferentes, isto porque o actor tem a tendência de exagerar a emoção. Com este trabalho foram criados corpora originais que serão um recurso importante a disponibilizar para futuras análises numa área que é ainda deficitária, em termos de investigação científica, em Portugal. Tanto os corpora, como respectivos resultados obtidos poderão vir a ser úteis em áreas como as Ciências da Fala; Robótica e Docência.

Do grafema ao gesto : contributos linguísticos para um sistema de síntese de base articulatória

Motivados pelo propósito central de contribuir para a construção, a longo prazo, de um sistema completo de conversão de texto para fala, baseado em síntese articulatória, desenvolvemos um modelo linguístico para o português europeu (PE), com base no sistema TADA (TAsk Dynamic Application), que visou a obtenção automática da trajectória dos articuladores a partir do texto de entrada. A concretização deste objectivo ditou o desenvolvimento de um conjunto de tarefas, nomeadamente 1) a implementação e avaliação de dois sistemas de silabificação automática e de transcrição fonética, tendo em vista a transformação do texto de entrada num formato adequado ao TADA; 2) a criação de um dicionário gestual para os sons do PE, de modo a que cada fone obtido à saída do conversor grafema-fone pudesse ter correspondência com um conjunto de gestos articulatórios adaptados para o PE; 3) a análise do fenómeno da nasalidade à luz dos princípios dinâmicos da Fonologia Articulatória (FA), com base num estudo articulatório e perceptivo. Os dois algoritmos de silabificação automática implementados e testados fizeram apelo a conhecimentos de natureza fonológica sobre a estrutura da sílaba, sendo o primeiro baseado em transdutores de estados finitos e o segundo uma implementação fiel das propostas de Mateus & d'Andrade (2000). O desempenho destes algoritmos – sobretudo do segundo – mostrou-se similar ao de outros sistemas com as mesmas potencialidades. Quanto à conversão grafema-fone, seguimos uma metodologia baseada em regras de reescrita combinada com uma técnica de aprendizagem automática. Os resultados da avaliação deste sistema motivaram a exploração posterior de outros métodos automáticos, procurando também avaliar o impacto da integração de informação silábica nos sistemas. A descrição dinâmica dos sons do PE, ancorada nos princípios teóricos e metodológicos da FA, baseou-se essencialmente na análise de dados de ressonância magnética, a partir dos quais foram realizadas todas as medições, com vista à obtenção de parâmetros articulatórios quantitativos. Foi tentada uma primeira validação das várias configurações gestuais propostas, através de um pequeno teste perceptual, que permitiu identificar os principais problemas subjacentes à proposta gestual. Este trabalho propiciou, pela primeira vez para o PE, o desenvolvimento de um primeiro sistema de conversão de texto para fala, de base articulatória. A descrição dinâmica das vogais nasais contou, quer com os dados de ressonância magnética, para caracterização dos gestos orais, quer com os dados obtidos através de articulografia electromagnética (EMA), para estudo da dinâmica do velo e da sua relação com os restantes articuladores. Para além disso, foi efectuado um teste perceptivo, usando o TADA e o SAPWindows, para avaliar a sensibilidade dos ouvintes portugueses às variações na altura do velo e alterações na coordenação intergestual. Este estudo serviu de base a uma interpretação abstracta (em termos gestuais) das vogais nasais do PE e permitiu também esclarecer aspectos cruciais relacionados com a sua produção e percepção.

Cegueira congénita na construção da realidade biofísica e psicossocial

O presente estudo teve como objetivo geral comparar a representação mental da realidade em crianças cegas congénitas, com a construção mental da realidade em crianças videntes, ao frequentarem o mesmo contexto de aprendizagem no Ensino Básico da Escola Regular (EBER). Esta comparação visou os seguintes objetivos específicos: (i) caracterizar as representações mentais construídas pelas crianças cegas congénitas a frequentar o EBER, (ii) comparar as representações mentais construídas pelas crianças cegas congénitas com as representações mentais construídas pelas crianças videntes, (iii) caracterizar as representações mentais que as crianças cegas congénitas constroem acerca da sua integração no EBER e (iv) caracterizar as representações mentais que os alunos videntes constroem acerca da integração das crianças cegas no EBER. O enquadramento teórico centrou-se nos conceitos de cegueira, desenvolvimento infantil e representações mentais. Metodologicamente, optámos por um design de estudos de caso múltiplos, com múltiplas unidades de análise. Para a recolha de dados recorremos a (i) entrevistas, (ii) conversas informais, (iii) questionário sociométrico e (iv) análise documental. Os resultados sugerem (i) ausência de diferenças significativas entre o grupo de sujeitos cegos congénitos e o grupo de videntes na identificação de estímulos de natureza percetual, (ii) ausência de diferenças significativas na riqueza, na complexidade e no total, entre as representações mentais evocadas por cegos congénitos e as representações mentais evocadas por videntes, (iii) ausência de diferenças significativas na natureza das informações entre as representações mentais evocadas por cegos congénitos e as representações mentais evocadas por videntes, (iv) ausência de diferenças significativas entre cegos congénitos e videntes, no número de preferências recebidas, no valor relativo tendo em consideração as ordens das preferências recebidas e no número de preferências recíprocas, (v) os videntes emitiram significativamente mais preferências que os respetivos pares cegos congénitos e (vi) o número de preferências emitidas pelos cegos congénitos está significativa e inversamente relacionado com as diferenças entre a riqueza, a complexidade e o total das representações mentais de cegos congénitos e videntes.

A performance do silêncio na música contemporânea para oboé

O presente estudo tem como objectivo entender todo o universo anexo à performance do silêncio na música clássica contemporânea. Numa nova perspectiva: o estudo do impacto da ansiedade num músico que se observa e que observa a sua interação com o contexto artístico que o rodeia, são estudadas todas as variáveis com influência directa no acto de percepção e gestão dos momentos de ausência de som. Tendo por base os princípios da investigação qualitativa foi criado um modelo teórico de análise na área dos estudos em performance. Este modelo foi aplicado em dois recitais realizados na Universidade de Aveiro e num concerto do Remix Ensemble da Casa da Música do Porto. O uso de ferramentas de pesquisa: O teste do Cortisol e o uso do colete Vitaljacket®, permitiu observar comportamentos fisiológicos no performer e no público nunca antes relatados. Os principais resultados desta investigação permitem aferir a elevada complexidade do fenómeno perceptivo e performativo do silêncio na música contemporânea. A pesquisa realizada permitiu observar que o silêncio na música contemporânea acontece num contexto de elevada ambiguidade semântica e que a duração dada a este num concerto, está directamente dependente do quadro fisiológico observado no performer no momento do concerto. Foi ainda possível aferir que a repetição, numa data posterior, de um mesmo programa de concerto provoca uma acentuada descida dos níveis de ansiedade no performer e essa descida altera a forma como este gere, em concerto, os momentos de silêncio. As conclusões alcançadas em sede de investigação reforçam a pertinência deste estudo. São postas à prova concepções altamente empíricas sobre a performance dos silêncios em música contemporânea. As asserções formuladas têm impacto a nível pedagógico e didático. São apontadas direcções de investigação futura no sentido da valorização do estudo da gestão dos momentos de silêncio em performance.

Grounding human vocabulary in robot perception through interaction

This thesis addresses the problem of word learning in computational agents. The motivation behind this work lies in the need to support language-based communication between service robots and their human users, as well as grounded reasoning using symbols relevant for the assigned tasks. The research focuses on the problem of grounding human vocabulary in robotic agent’s sensori-motor perception. Words have to be grounded in bodily experiences, which emphasizes the role of appropriate embodiments. On the other hand, language is a cultural product created and acquired through social interactions. This emphasizes the role of society as a source of linguistic input. Taking these aspects into account, an experimental scenario is set up where a human instructor teaches a robotic agent the names of the objects present in a visually shared environment. The agent grounds the names of these objects in visual perception. Word learning is an open-ended problem. Therefore, the learning architecture of the agent will have to be able to acquire words and categories in an openended manner. In this work, four learning architectures were designed that can be used by robotic agents for long-term and open-ended word and category acquisition. The learning methods used in these architectures are designed for incrementally scaling-up to larger sets of words and categories. A novel experimental evaluation methodology, that takes into account the openended nature of word learning, is proposed and applied. This methodology is based on the realization that a robot’s vocabulary will be limited by its discriminatory capacity which, in turn, depends on its sensors and perceptual capabilities. An extensive set of systematic experiments, in multiple experimental settings, was carried out to thoroughly evaluate the described learning approaches. The results indicate that all approaches were able to incrementally acquire new words and categories. Although some of the approaches could not scale-up to larger vocabularies, one approach was shown to learn up to 293 categories, with potential for learning many more.

Psycho-physio dynamics in violin-piano duo: a pianist's perspective

The case study looked at psychological and physiological responses to stress in musicians, comparing a newly formed and a consolidated violin-piano duo. The common element between these duos was the pianist. Using the psychological tests (STAI Y1 and Y2, K-MPAI, MMPI-2, ICAC), the immunoassay saliva test to measure cortisol (stress hormone) and non- invasive device VitalJacket® developed at the University of Aveiro, Portugal, participants were monitored under various performance conditions. Others quantitative and qualitative dataset were collected including a pianist’s personal diary (analyzed by psychiatrist), semi-structured interviews with members of long-terms chamber music duo and perceptual evaluations (listening test) of the performances by expert listeners. The variables included two performance venues (European university and secondary school), as well as well-known repertoire, recently known repertoire and newly known repertoire. The latter was given approximately one week before each recital. The psychological and physiological dataset were collected for a total of eight recitals – two series of four recitals each. The unexpected results show that state anxiety levels and stress of the pianist, who does not present an anxious profile, either in social or in musical terms, are always higher when playing with a well-known partner. Possible explanations may be due to the highest expectations for quality of performance and implications of mirror neurons (since the reactions are very different according to the partner). In other words, the “known” (i.e., the consolidated duo) can become “trapped” within a predetermined space, especially at the psychological level, while the “unknown” (the occasional duo) seems to be less involved and therefore more reassuring and exciting in positive terms. In addition, the preference of the expert audience is for the consolidated duo.

In search of reliable centimeter-level indoor positioning: a smartphone-based approach

This thesis describes the design and implementation of a reliable centimeter-level indoor positioning system fully compatible with a conventional smartphone. The proposed system takes advantage of the smartphone audio I/O and processing capabilities to perform acoustic ranging in the audio band using non-invasive audio signals and it has been developed having in mind applications that require high accuracy, such as augmented reality, virtual reality, gaming and audio guides. The system works in a distributed operation mode, i.e. each smartphone is able to obtain its own position using only acoustic signals. To support the positioning system, a Wireless Sensor Network (WSN) of synchronized acoustic beacons is used. To keep the infrastructure in sync we have developed an Automatic Time Synchronization and Syntonization (ATSS) protocol with a standard deviation of the sync offset error below 1.25 μs. Using an improved Time Difference of Arrival (TDoA) estimation approach (which takes advantage of the beacon signals’ periodicity) and by performing Non-Line-of-Sight (NLoS) mitigation, we were able to obtain very stable and accurate position estimates with an absolute mean error of less than 10 cm in 95% of the cases and a mean standard deviation of 2.2 cm for a position refresh period of 350 ms.

Between the Bullet and the Hole

Between the Bullet and the Hole is a film centred on the elusive and complex effects of war on women's role in ballistic research and early computing. The film features new and archival high-speed bullet photography, schlieren and electric spark imagery, bullet sound wave imagery, forensic ballistic photography, slide rulers, punch cards, computer diagrams, and a soundtrack by Scanner. Like a frantic animation storyboard, it explores the flickering space between the frames, testing the perceptual mechanics of visual interpolation, the possibility of reading or deciphering the gap between before and after. Interpolation - the main task of the women studying ballistics in WW2 - is the construction or guessing of missing data using only two known data points. The film tries to unpack this gap, open it up to interrogation. It questions how we read, interpolate or construct the gaps between bullet and hole, perpetrator and victim, presence and absence. The project involves exchanges with specialists in this area such as the Cranfield University Forensics department, London-based Forensic Firearms consultancy, the Imperial War Museum, the ENIAC programmers project, the Smithsonian Institute, and Forensic Scientists at Palm Beach County Sheriff's Office (USA). Exhibitions: Solo exhibition at Dallas Contemporary (Texas, Jan-Mar 2016), including newly commissioned lenticular prints and a dual slide projector installation; Group exhibition the Sydney Biennale (Sydney, Mar-June 2016); UK premiere and solo retrospective screening at Whitechapel Gallery (London); forthcoming solo exhibition at Iliya Fridman Gallery (NY, Oct-Dec 2016); Film festivals and screenings: International Film Festival Rotterdam (Jan 2016); Whitechapel Gallery (London Feb 2016); Cornerhouse/Home (Manchester Nov 2016); Public lectures: Whitechapel Gallery with prof. David Alan Grier and Morgan Quaintance; Carriageworks (Sydney) Prof. Douglas Khan; Monash University (Melbourne); Gertrude Space (Melbourne). Reviews and interviews: Artforum, Studio International, Mousse Magazine.

Face matching impairment in developmental prosopagnosia

Developmental prosopagnosia (DP) is commonly referred to as ‘face blindness’, a term that implies a perceptual basis to the condition. However, DP presents as a deficit in face recognition and is diagnosed using memory-based tasks. Here, we test face identification ability in six people with DP, who are severely impaired on face memory tasks, using tasks that do not rely on memory. First, we compared DP to control participants on a standardised test of unfamiliar face matching using facial images taken on the same day and under standardised studio conditions (Glasgow Face Matching Test; GFMT). DP participants did not differ from normative accuracy scores on the GFMT. Second, we tested face matching performance on a test created using images that were sourced from the Internet and so vary substantially due to changes in viewing conditions and in a person’s appearance (Local Heroes Test; LHT). DP participants show significantly poorer matching accuracy on the LHT relative to control participants, for both unfamiliar and familiar face matching. Interestingly, this deficit is specific to ‘match’ trials, suggesting that people with DP may have particular difficulty in matching images of the same person that contain natural day-to-day variations in appearance. We discuss these results in the broader context of individual differences in face matching ability.