MULTIMODAL PERFORMANCE AND RELIABILITY ANALYSIS OF THE PIERRE AUGER OBSERVATORY NORTH SITE

The Pierre Auger Cosmic Ray Observatory North site employs a large array of surface detector stations (tanks) to detect the secondary particle showers generated by ultra-high energy cosmic rays. Due to the rare nature of ultra-high energy cosmic rays, it is important to have a high reliability on tank communications, ensuring no valuable data is lost. The Auger North site employs a peer-to-peer paradigm, the Wireless Architecture for Hard Real-Time Embedded Networks (WAHREN), designed specifically for highly reliable message delivery over fixed networks, under hard real-time deadlines. The WAHREN design included two retransmission protocols, Micro- and Macro- retransmission. To fully understand how each retransmission protocol increased the reliability of communications, this analysis evaluated the system without using either retransmission protocol (Case-0), both Micro- and Macro-retransmission individually (Micro and Macro), and Micro- and Macro-retransmission combined. This thesis used a multimodal modeling methodology to prove that a performance and reliability analysis of WAHREN was possible, and provided the results of the analysis. A multimodal approach was necessary because these processes were driven by different mathematical models. The results from this analysis can be used as a framework for making design decisions for the Auger North communication system.

A PHENOMENOLOGY OF MIMETIC LEARNING AND MULTIMODAL COGNITION: INTEGRATING EXPERIENTIAL KNOWLEDGE INTO PROGRAMS IN RHETORIC, COMPOSITION, AND TECHNICAL COMMUNICATION

My dissertation emphasizes a cognitive account of multimodality that explicitly integrates experiential knowledge work into the rhetorical pedagogy that informs so many composition and technical communication programs. In these disciplines, multimodality is widely conceived in terms of what Gunther Kress calls “socialsemiotic” modes of communication shaped primarily by culture. In the cognitive and neurolinguistic theories of Vittorio Gallese and George Lakoff, however, multimodality is described as a key characteristic of our bodies’ sensory-motor systems which link perception to action and action to meaning, grounding all communicative acts in knowledge shaped through body-engaged experience. I argue that this “situated” account of cognition – which closely approximates Maurice Merleau-Ponty’s phenomenology of perception, a major framework for my study – has pedagogical precedence in the mimetic pedagogy that informed ancient Sophistic rhetorical training, and I reveal that training’s multimodal dimensions through a phenomenological exegesis of the concept mimesis. Plato’s denigration of the mimetic tradition and his elevation of conceptual contemplation through reason, out of which developed the classic Cartesian separation of mind from body, resulted in a general degradation of experiential knowledge in Western education. But with the recent introduction into college classrooms of digital technologies and multimedia communication tools, renewed emphasis is being placed on the “hands-on” nature of inventive and productive praxis, necessitating a revision of methods of instruction and assessment that have traditionally privileged the acquisition of conceptual over experiential knowledge. The model of multimodality I construct from Merleau-Ponty’s phenomenology, ancient Sophistic rhetorical pedagogy, and current neuroscientific accounts of situated cognition insists on recognizing the significant role knowledges we acquire experientially play in our reading and writing, speaking and listening, discerning and designing practices.

Effects of modality, urgency and situation on responses to multimodal warnings for drivers

Signifying road-related events with warnings can be highly beneficial, especially when imminent attention is needed. This thesis describes how modality, urgency and situation can influence driver responses to multimodal displays used as warnings. These displays utilise all combinations of audio, visual and tactile modalities, reflecting different urgency levels. In this way, a new rich set of cues is designed, conveying information multimodally, to enhance reactions during driving, which is a highly visual task. The importance of the signified events to driving is reflected in the warnings, and safety-critical or non-critical situations are communicated through the cues. Novel warning designs are considered, using both abstract displays, with no semantic association to the signified event, and language-based ones, using speech. These two cue designs are compared, to discover their strengths and weaknesses as car alerts. The situations in which the new cues are delivered are varied, by simulating both critical and non-critical events and both manual and autonomous car scenarios. A novel set of guidelines for using multimodal driver displays is finally provided, considering the modalities utilised, the urgency signified, and the situation simulated.

Efeitos de um programa de exercício multimodal em pessoas idosas institucionalizadas

Objetivo: Investigar os efeitos de um programa de exercício multimodal sobre o funcionamento cognitivo e aptidão física funcional em pessoas idosas institucionalizadas. Método: Os participantes foram selecionados por conveniência entre os utentes de duas instituições de apoio a pessoas idosas. O grupo inicial foi constituído por um total de 21 pessoas de ambos os sexos (77-92 anos). Inicialmente, foi garantido um “período de controlo” de 4 semanas, em que os participantes mantiveram as suas atividades de vida quotidianas normais. Posteriormente, todos os idosos integraram um programa de exercício multimodal durante 8 semanas. Para estudar os efeitos da intervenção foram recolhidos dados em 3 momentos distintos: previamente ao período de controlo, após o período de controlo e no final do programa de intervenção. Foram efetuados testes físicos/motores, cognitivos e de dupla-tarefa (motor-cognitivo). Durante as sessões de exercício, alternaram-se períodos constituídos por tarefas motoras com períodos em que as tarefas motoras implicavam uma mobilização simultânea de recursos cognitivos. Resultados: A análise estatística dos dados recolhidos nos três momentos de avaliação, revelou efeitos positivos da intervenção ao nível da atenção, tempo de reação, força muscular, agilidade e capacidade cardiorrespiratória. Nos testes realizados em dupla-tarefa, foram encontradas melhorias no teste timed up and go, mas não se verificaram melhorias no teste de tempo de reação. A análise estatística dos dados recolhidos nos três momentos de avaliação, revelou efeitos positivos (p<0.05). da intervenção ao nível da atenção, tempo de reação, força muscular, agilidade e capacidade cardiorrespiratória. Nos testes realizados em dupla-tarefa, foram encontradas melhorias no teste timed up and go (p<0.05), mas não se verificaram melhorias no teste de tempo de reação. Conclusões: Um programa de exercícios multimodal induz melhorias no funcionamento cognitivo e motor de pessoas idosas institucionalizadas. É aconselhada a divulgação deste tipo de intervenção para pessoas idosas institucionalizadas; Effects of a Multimodal Exercise Program for Elderly Institutionalized Abstract: Objective: To investigate wthe effects of a multimodal exercise program on cognitive functioning and physical fitness of institutionalized elderly. Method: Participants were selected by convenience among two nursing home residents. Twenty-one people (77-92 years old) of both genders participated. During the first 4 weeks (control period), participants were not engaged in the exercise program and continue with their normal daily life activities. After the control period, the group was engaged in a multimodal exercise program for 8 weeks (2 times per week). To study the effects of a multimodal exercise program, data were collected at three different times: prior to the control period, after the control period, and at the end of the intervention. Several motor tests (physical fitness), cognitive tests and dual-tasks (motor-cognitive) were performed. The exercise sessions alternated between periods of motor taks and periods with motor plus cognitive tasks performed simultaneously. Results: The analysis of the data collected in the three moments of evaluation, show positive effects of the exercise program in cognitive dimensions (information processing speed and attention) and functional physical fitness components (muscle resistance, cardiorespiratory fitness and dynamic balance). In tests carried out in dual-task conditions, the exercise program promoted improvements in the timed up and go test (with mental calculations). Statistical analysis of the data collected at the three evaluation moments revealed positive effects (p <0.05). Of attention intervention, reaction time, muscle strength, agility and cardiorespiratory capacity. In the double-task tests, improvements were found in the timed up and go test (p <0.05), but there were no improvements in the reaction time test. Conclusions: The results of this study indicate that a multimodal exercise program can improve cognitive functioning and physical fitness in institutionalized older people. Thus, this type of intervention should be promoted among nursing home residents.

The Analogy as a Cognitive Strategy that Encourages Reading Comprehension in Expository Texts

This research arises from the current state of reading in Mexico, because the contextual situation to local and national level has become critical on this issue. Results on standardized tests such as ENLACE (National Assessment of Academic Achievement in Schools) and PISA (Program for International Student Assessment) have shown that Mexican basic level students have text comprehension problems. Given this reality, this is a qualitative descriptive study on the use of analogies in reading as a strategy for reading comprehension, through application of an ethnographic survey, a questionnaire and the application of a pedagogical model for the use of analogies in reading. The study incorporates the theoretical tenets of Constructivism, referring to Piaget (1969), Vygotsky (1997) and Ausubel (2002). The results show that the use of analogies promotes reading comprehension in 4th grade students, from an expository text that compares the evolution of man and the metamorphosis of a butterfly.

Plagiarism in Academic Texts

The ethical and social responsibility of citing the sources in a scientific or artistic work is undeniable. This paper explores, in a preliminary way, academic plagiarism in its various forms. It includes findings based on a forensic analysis. The purpose of this paper is to raise awareness on the importance of considering these details when writing and publishing a text. Hopefully, this analysis may put the issue under discussion.

Language, action, and sensoriality: a multimodal analysis of interactions in inclusive sport climbing with visually impaired athletes

In sport climbing, athletes with vision impairments are constantly accompanied by their guides – usually trainers – both during the preparatory inspection of the routes and whilst climbing. Trainers are, so to speak, the climbers’ eyes, in the sense that they systematically put their vision in the service of the climbers’ mobility and sporting performance. The synergy between trainers and athletes is based on peculiar, strictly multimodal interactive practices that are focused on the body and on its constantly evolving sensory engagement with the materiality of routes. In this context, sensory perception and embodied actions required to plan and execute the climb are configured as genuinely interactive accomplishments. Drawing on the theoretical framework of Embodied and Situated Cognition and on the methodology of Conversation Analysis, this thesis engages in the multimodal analysis of trainer-athlete interactions in paraclimbing. The analysis is based on a corpus of video recorded climbing sessions. The major findings of the study can be summarized as follows. 1) Intercorporeality is key to interactions between trainers and athletes with visual impairments. The participants orient to perceiving the climbing space and acting in it as a ‘We’. 2) The grammar, lexicon, prosody, and timing of the trainers’ instructions are finely tuned to the ongoing corporeal experience of the climbers. 3) Climbers with visual impairments build their actions by using sensory resources that are provided by their trainers. This result is of particular importance as it shows that resources and constraints for action are in a fundamental way constituted in interaction with Others and with specific socio-material ecologies, rather than being defined a priori by the organs and functions of individuals’ body and mind. Individual capabilities are thus enhanced and extended in interaction, which encourages a more ecological view of (dis)ability.

Typing your mind away. Comparing keylogged tasks with the Task Segment Framework

One of the main process features under study in Cognitive Translation & Interpreting Studies (CTIS) is the chronological unfolding of the tasks. The analyses of time spans in translation have been conceived in two ways: (1) studying those falling between text units of different sizes: words, phrases, sentences, and paragraphs; (2) setting arbitrary time span thresholds to explore where do they fall in the text, whether between text units or not. Writing disfluencies may lead to comprehensive insights into the cognitive activities involved in typing while translating. Indeed, long time spans are often taken as hints that cognitive resources have been subtracted from typing and devoted to other activities, such as planning, evaluating, etc. This exploratory, pilot study combined both approaches to seek potential general tendencies and contrasts in informants’ inferred mental processes when performing different writing tasks, through the analysis of their behaviors, as keylogged. The study tasks were retyping, monolingual free writing, translation, revision and a multimodal task—namely, monolingual text production based on an infographic leaflet. Task logs were chunked, and shorter time spans, including those within words, were analyzed following the Task Segment Framework (Muñoz & Apfelthaler, in press). Finally, time span analysis was combined with the analysis of the texts as to their lexical density, type-token ratio and word frequency. Several previous results were confirmed, and some others were surprising. Time spans in free writing were longer between paragraphs and sentences, possibly hinting at planning and, in translation, between clauses and words, suggesting more cognitive activities at these levels. On the other hand, the infographic was expected to facilitate the writing process, but most time spans were longer than in both free writing and translation. Results of the multimodal task and some other results suggest venues for further research.

Print versus digital texts: understanding the experimental research and challenging the dichotomies

This article presents the results of a systematic critical review of interdisciplinary literature concerned with digital text (or e-text) uses in education and proposes recommendations for how e-texts can be implemented for impactful learning. A variety of e-texts can be found in the repertoire of educational resources accessible to students, and in the constantly changing terrain of educational technologies, they are rapidly evolving, presenting new opportunities and affordances for student learning. We highlight some of the ways in which academic studies have examined e-texts as part of teaching and learning practices, placing a particular emphasis on aspects of learning such as recall, comprehension, retention of information and feedback. We also review diverse practices associated with uses of e-text tools such as note-taking, annotation, bookmarking, hypertexts and highlighting. We argue that evidence-based studies into e-texts are overwhelmingly structured around reinforcing the existing dichotomy pitting print-based (‘traditional’) texts against e-texts. In this article, we query this approach and instead propose to focus on factors such as students’ level of awareness of their options in accessing learning materials and whether they are instructed and trained in how to take full advantage of the capabilities of e-texts, both of which have been found to affect learning performance.

Hybrid Artificial Intelligence to Extract Patterns and Rules from Argumentative and Legal Texts

This Thesis is composed of a collection of works written in the period 2019-2022, whose aim is to find methodologies of Artificial Intelligence (AI) and Machine Learning to detect and classify patterns and rules in argumentative and legal texts. We define our approach “hybrid”, since we aimed at designing hybrid combinations of symbolic and sub-symbolic AI, involving both “top-down” structured knowledge and “bottom-up” data-driven knowledge. A first group of works is dedicated to the classification of argumentative patterns. Following the Waltonian model of argument and the related theory of Argumentation Schemes, these works focused on the detection of argumentative support and opposition, showing that argumentative evidences can be classified at fine-grained levels without resorting to highly engineered features. To show this, our methods involved not only traditional approaches such as TFIDF, but also some novel methods based on Tree Kernel algorithms. After the encouraging results of this first phase, we explored the use of a some emerging methodologies promoted by actors like Google, which have deeply changed NLP since 2018-19 — i.e., Transfer Learning and language models. These new methodologies markedly improved our previous results, providing us with best-performing NLP tools. Using Transfer Learning, we also performed a Sequence Labelling task to recognize the exact span of argumentative components (i.e., claims and premises), thus connecting portions of natural language to portions of arguments (i.e., to the logical-inferential dimension). The last part of our work was finally dedicated to the employment of Transfer Learning methods for the detection of rules and deontic modalities. In this case, we explored a hybrid approach which combines structured knowledge coming from two LegalXML formats (i.e., Akoma Ntoso and LegalRuleML) with sub-symbolic knowledge coming from pre-trained (and then fine-tuned) neural architectures.

Investigating interpreter-mediated interaction in the Chinese bilingual courtroom: Analysis based on a multimodal corpus

This research project is based on the Multimodal Corpus of Chinese Court Interpreting (MUCCCI [mutʃɪ]), a small-scale multimodal corpus on the basis of eight authentic court hearings with Chinese-English interpreting in Mainland China. The corpus has approximately 92,500 word tokens in total. Besides the transcription of linguistic and para-linguistic features, utilizing the facial expression classification rules suggested by Black and Yacoob (1995), MUCCCI also includes approximately 1,200 annotations of facial expressions linked to the six basic types of human emotions, namely, anger, disgust, happiness, surprise, sadness, and fear (Black & Yacoob, 1995). This thesis is an example of conducting qualitative analysis on interpreter-mediated courtroom interactions through a multimodal corpus. In particular, miscommunication events (MEs) and the reasons behind them were investigated in detail. During the analysis, although queries were conducted based on non-verbal annotations when searching for MEs, both verbal and non-verbal features were considered indispensable parts contributing to the entire context. This thesis also includes a detailed description of the compilation process of MUCCCI utilizing ELAN, from data collection to transcription, POS tagging and non-verbal annotation. The research aims at assessing the possibility and feasibility of conducting qualitative analysis through a multimodal corpus of court interpreting. The concept of integrating both verbal and non-verbal features to contribute to the entire context is emphasized. The qualitative analysis focusing on MEs can provide an inspiration for improving court interpreters’ performances. All the constraints and difficulties presented can be regarded as a reference for similar research in the future.