The Validity and Comparison of Two Affiliation Tasks in Schizophrenia

Negative symptoms are related to worse psychosocial functioning in schizophrenia. The current study evaluates two behavioral affiliation tasks—the video-based Social Affiliation Interaction Task (SAIT) and the in-vivo Conversation Task (CT)—and explores whether behavioral ratings of social affiliation are associated with negative symptoms and community functioning. Participants, 20 with schizophrenia/schizoaffective disorder (SZ) and 35 healthy controls (HC), completed both tasks and measures of negative symptoms and functioning. SZ evidenced lower behavioral affiliation on the SAIT compared to HC. There were no group differences in behavioral affiliation on the CT. Within groups, behavioral affiliation was not correlated between tasks or with symptoms and functioning. Across groups, behavioral affiliation from the SAIT was correlated with symptoms and functioning. Post hoc analyses revealed higher ratings of positive facial expression and valence in the CT for HC compared to SZ. Results suggest that the method of assessing behavioral affiliaton may influence research findings.

Interaction techniques with novel multimodal feedback for addressing gesture-sensing systems

Users need to be able to address in-air gesture systems, which means finding where to perform gestures and how to direct them towards the intended system. This is necessary for input to be sensed correctly and without unintentionally affecting other systems. This thesis investigates novel interaction techniques which allow users to address gesture systems properly, helping them find where and how to gesture. It also investigates audio, tactile and interactive light displays for multimodal gesture feedback; these can be used by gesture systems with limited output capabilities (like mobile phones and small household controls), allowing the interaction techniques to be used by a variety of device types. It investigates tactile and interactive light displays in greater detail, as these are not as well understood as audio displays. Experiments 1 and 2 explored tactile feedback for gesture systems, comparing an ultrasound haptic display to wearable tactile displays at different body locations and investigating feedback designs. These experiments found that tactile feedback improves the user experience of gesturing by reassuring users that their movements are being sensed. Experiment 3 investigated interactive light displays for gesture systems, finding this novel display type effective for giving feedback and presenting information. It also found that interactive light feedback is enhanced by audio and tactile feedback. These feedback modalities were then used alongside audio feedback in two interaction techniques for addressing gesture systems: sensor strength feedback and rhythmic gestures. Sensor strength feedback is multimodal feedback that tells users how well they can be sensed, encouraging them to find where to gesture through active exploration. Experiment 4 found that they can do this with 51mm accuracy, with combinations of audio and interactive light feedback leading to the best performance. Rhythmic gestures are continuously repeated gesture movements which can be used to direct input. Experiment 5 investigated the usability of this technique, finding that users can match rhythmic gestures well and with ease. Finally, these interaction techniques were combined, resulting in a new single interaction for addressing gesture systems. Using this interaction, users could direct their input with rhythmic gestures while using the sensor strength feedback to find a good location for addressing the system. Experiment 6 studied the effectiveness and usability of this technique, as well as the design space for combining the two types of feedback. It found that this interaction was successful, with users matching 99.9% of rhythmic gestures, with 80mm accuracy from target points. The findings show that gesture systems could successfully use this interaction technique to allow users to address them. Novel design recommendations for using rhythmic gestures and sensor strength feedback were created, informed by the experiment findings.

A flexible bio-inspired hierarchical model for analyzing musical timbre

A flexible and multipurpose bio-inspired hierarchical model for analyzing musical timbre is presented in this paper. Inspired by findings in the fields of neuroscience, computational neuroscience, and psychoacoustics, not only does the model extract spectral and temporal characteristics of a signal, but it also analyzes amplitude modulations on different timescales. It uses a cochlear filter bank to resolve the spectral components of a sound, lateral inhibition to enhance spectral resolution, and a modulation filter bank to extract the global temporal envelope and roughness of the sound from amplitude modulations. The model was evaluated in three applications. First, it was used to simulate subjective data from two roughness experiments. Second, it was used for musical instrument classification using the k-NN algorithm and a Bayesian network. Third, it was applied to find the features that characterize sounds whose timbres were labeled in an audiovisual experiment. The successful application of the proposed model in these diverse tasks revealed its potential in capturing timbral information.

This Is a Gorgeous-looking Movie”: Multimodal Interplay between Commentary and Subject Materials in Movie Review Entertainment

The present thesis is a study of movie review entertainment (MRE) which is a contemporary Internet-based genre of texts. MRE are movie reviews in video form which are published online, usually as episodes of an MRE web show. Characteristic to MRE is combining humor and honest opinions in varying degrees as well as the use of subject materials, i.e. clips of the movies, as a part of the review. The study approached MRE from a linguistic perspective aiming to discover 1) whether MRE is primarily text- or image-based and what the primary functions of the modes are, 2) how a reviewer linguistically combines subject footage to her/his commentary?, 3) whether there is any internal variation in MRE regarding the aforementioned questions, and 4) how suitable the selected models and theories are in the analysis of this type of contemporary multimodal data. To answer the aforementioned questions, the multimodal system of image—text relations by Martinec and Salway (2005) in combination with categories of cohesion by Halliday and Hasan (1976) were applied to four full MRE videos which were transcribed in their entirety for the study. The primary data represent varying types of MRE: a current movie review, an analytic essay, a riff review, and a humorous essay. The results demonstrated that image vs. text prioritization can vary between reviews and also within a review. The current movie review and the two essays were primarily commentary-focused whereas the riff review was significantly more dependent on the use of imagery as the clips are a major source of humor which is a prominent value in that type of a review. In addition to humor, clips are used to exemplify the commentary. A reviewer also relates new information to the imagery as well as uses two modes to present the information in a review. Linguistically, the most frequent case was that the reviewer names participants and processes lexically in the commentary. Grammatical relations (reference items such as pronouns and adverbs and conjunctive items in the riff review) were also encountered. There was internal variation to a considerable degree. The methods chosen were deemed appropriate to answer the research questions. Further study could go beyond linguistics to include, for instance, genre and media studies.

El papel de la gestualidad en la estructuraci?n multimodal del discurso [recurso electr?nico]

Durante los ?ltimos a?os el an?lisis del discurso oral ha tomado un lugar importante en los estudios de la ling??stica y sobretodo en investigaciones enfocadas en el discurso en el aula. Su naturaleza en el habla permite obtener un gran abanico de elementos por analizar que marca la diferencia con el an?lisis del discurso escrito. Por otro lado, en la comunicaci?n oral el hablante est? pendiente de lo que dice con las palabras, pero no controla de la misma manera los gestos. Aspectos como la expresi?n del rostro, la orientaci?n corporal, la direcci?n de la mirada, etc., enfatizan lo que se quiere expresar en el discurso verbal. As?, los enunciados son bimodales ya que utilizan tanto la modalidad auditivo-vocal como la viso-gestual. Sin embargo, este estudio caracteriza este tipo de comunicaci?n on l t rm no ?mult mo l? o l o qu n l s urso or l l orpus que se analiza hay un canal extra de comunicaci?n como lo es el uso del tablero. Esta exploraci?n le da valor a los gestos en el ?mbito pedag?gico, pues el lenguaje no verbal puede reforzar o sustituir la expresi?n verbal y puede llegar a tener una gran injerencia en la construcci?n del sentido que los alumnos elaboran en torno a la comunicaci?n transmitida por el profesor. Teniendo en cuenta que la lecci?n es una unidad textual, esta investigaci?n busca observar y analizar c?mo los gestos contribuyen a la estructuraci?n del discurso en el aula, es decir, cu?l ser?a la marcaci?n multimodal de la estructura de la lecci?n. Se exploran dos temas centrales: el an?lisis del discurso en el aula y la gestualidad que acompa?a al habla. Para ello se utiliz? el modelo de an?lisis del discurso oral de Sinclair y Coulthard (1992) que tiene como elementos principales la lecci?n, las transacciones y los intercambios, y el modelo de an?lisis de gesticulaciones de McNeill (1998, 2005) que propone cinco dimensiones de gesticulaciones: gestos r?tmicos, gestos de?cticos, gestos ic?nicos, gestos metaf?ricos y gestos cohesivos.Se busc? determinar el papel de los gestos como marcadores o reforzadores de la estructura del discurso oral de una clase de Matem?ticas en ingl?s, dada por un docente nativo para quinto grado en un contexto educativo biling?e. Se hizo una grabaci?n de esa clase de 45 minutos, se procedi? a transcribirla en un formato de dos columnas: Profesor (con los enunciados del profesor) y Estudiantes (con las intervenciones de los estudiantes). Se observ? que el estudio del discurso oral en el aula se enriquece cuando se le a?aden elementos de an?lisis gestual en el mismo. Esta investigaci?n invita a indagar m?s sobre el papel de estos dos componentes de la interacci?n (discurso oral y gestualidad) tanto en el aula como en otras situaciones comunicativas

The impact of the integration of ict on elementary efl students’ vocabulary at the corporación unificada nacional cun through writing communicative tasks

Tesis (Licenciado en Lenguas Castellana, Inglés y Francés).--Universidad de La Salle. Facultad de Ciencias de La Educación. Licenciatura en Lengua Castellana, Inglés y Francés, 2014

Improving CEGEP Students’ ESL Accuracy through L1/L2 Synchronous Text-based Screen-Sharing Tasks

Abstract : CEGEPs are now reaping the ‘first fruits’ of the last Educational Reform in Quebec and as a result, ‘English as Second Language’ (ESL) teachers are noticing an improvement in fluency and a seemingly lower level of inhibition when it comes to production skills. However, this output is accompanied by a noticeable lack of accuracy. Keeping in mind that the purpose of language is communication, we need to find a way to reduce the number of basic common errors made by CEGEP ESL students, while maintaining a natural and motivating learning environment. Thanks to recent advances in computer-mediated communication (CMC), we now have the necessary tools to access peer native speakers throughout the world. Although this technology can be used for other language courses, this study explored the potential value of collaboration with native English speakers through the use of synchronous screen-sharing technology, in order to improve CEGEP ESL students’ accuracy in writing. The instrumentation used consisted of a questionnaire, tests, plus documents of collaborative tasks, using the ‘Google for Education’ screen-sharing tool. Fourteen Intermediate/Advanced ESL CEGEP students participated in this study. Despite the positive tendencies revealed, only a prolonged use of the innovative method yielded a significant positive impact. Moreover, a mixed linear regression for the group with more L1 intervention revealed a significant correlation between the number of errors in the task documents and the number of tasks accomplished. Thus, it could be inferred that ESL accuracy improves in proportion to the number of synchronous text-based screen-sharing tasks done with L1 collaboration.

Balancing the demands of two tasks: an investigation of cognitive-motor dual-tasking in relapsing remitting multiple sclerosis

Background: People with relapsing remitting MS (PwRRMS) suffer disproportionate decrements in gait under dual-task conditions, when walking and a cognitive task are combined. There has been much less investigation of the impact of cognitive demands on balance. This study investigated whether: (1) PwRRMS show disproportionate decrements in postural stability under dual-task conditions compared to healthy controls; (2) dual-task decrements are associated with everyday dual-tasking difficulties. In addition, the impact of mood, fatigue and disease severity on dual-tasking were also examined. Methods: 34 PwRRMS and 34 matched controls completed cognitive (digit span) and balance (movement of centre of pressure on a Biosway, on stable and unstable surfaces) tasks under single and dual-task conditions. Everyday dual-tasking was measured using the DTQ. Mood was measured by the HADS. Fatigue was measured via the MFIS. Results: No differences in age, gender, years of education, estimated pre-morbid IQ or baseline digit span between the groups. Compared to healthy controls, PwRRMS showed a significantly greater decrement in postural stability under dual-task conditions on an unstable surface (p=0.007), but not a stable surface (p=0.679). PwRRMS reported higher levels of everyday dual-tasking difficulties (p<0.001). Balance decrement scores were not correlated with everyday dual-tasking difficulties, or with fatigue. Stable surface balance decrement scores were significantly associated with levels of anxiety (rho=0.527, p=0.001) and depression (rho=0.451, p=0.007). Conclusion: RRMS causes difficulties with dual-tasking, impacting balance, particularly under challenging conditions, which may contribute to an increased risk of gait difficulties and falls. The striking relationship between anxiety/depression and dual-task decrement suggests that worry may be contributing to dual-task difficulties.

La imagen en el discurso multimodal cortazariano, una poética en movimiento

The selection and proposed cortazarian image in multimodal speech, a poetic moving as a doctoral thesis title, is the result of a process that forms several progressive stages. more than satisfy our curiosity and literary artistic concerns, the questions multiply increases our desire to bring more creative Cortazarian the colossus, that not only ruins a way of making literature , but in its revolutionary labyrinth gives a turn of the screw to the already tangled world of artistic writing , breaking in 1967 with the first of four hybrids books, La vuelta al día en ochenta mundos through which converts the traditional plastic unitextual and unimodal space in a palempsesto inexhaustible generator senses and meanings. Far from appeasing the wrath of conservative critics, caused by his narrative masterpiece, Hopscotch (1963), sparked the debate to denounce the regression of the classic molds of writing, proposing the imagination as the setting for creative freedom, it is not as arbitrary and nonsense, but as a process of higher state of consciousness in which it operates an underlying logic. Hence, our objective is to crawl into the underworld and plastics other multisígnicas border spaces, fleeting and ephemeral alliances that relate not just as complementary and similar texts which may be the same speech and can transmit the same, but quite the contrary, the oft-repeated notion of analogy is the first victim, fortunately, this kind of creative-artistic operations that raise the creative act to a sublime state capable of converting the inexhaustible multitextuales constellations in different ways: opposition, confrontation, invasion, dialogue, shadow, duplication, theft, etc. However this multiple transgression pushed to the limit by Julio Cortazar and his friend Julio Silva through four books proposed for analysis often does not translate into jobs and research in literature departments of the universities of the world, nor the critics are echoes the striking abundance and clarity of expression multimodal phenomenon as is the case with other works of Cortazar unimodal type, which makes it impassable fences mysterious worlds...

ViDRILO: The Visual and Depth Robot Indoor Localization with Objects information dataset

In this article we describe a semantic localization dataset for indoor environments named ViDRILO. The dataset provides five sequences of frames acquired with a mobile robot in two similar office buildings under different lighting conditions. Each frame consists of a point cloud representation of the scene and a perspective image. The frames in the dataset are annotated with the semantic category of the scene, but also with the presence or absence of a list of predefined objects appearing in the scene. In addition to the frames and annotations, the dataset is distributed with a set of tools for its use in both place classification and object recognition tasks. The large number of labeled frames in conjunction with the annotation scheme make this dataset different from existing ones. The ViDRILO dataset is released for use as a benchmark for different problems such as multimodal place classification and object recognition, 3D reconstruction or point cloud data compression.

MULTIMODAL PERFORMANCE AND RELIABILITY ANALYSIS OF THE PIERRE AUGER OBSERVATORY NORTH SITE

The Pierre Auger Cosmic Ray Observatory North site employs a large array of surface detector stations (tanks) to detect the secondary particle showers generated by ultra-high energy cosmic rays. Due to the rare nature of ultra-high energy cosmic rays, it is important to have a high reliability on tank communications, ensuring no valuable data is lost. The Auger North site employs a peer-to-peer paradigm, the Wireless Architecture for Hard Real-Time Embedded Networks (WAHREN), designed specifically for highly reliable message delivery over fixed networks, under hard real-time deadlines. The WAHREN design included two retransmission protocols, Micro- and Macro- retransmission. To fully understand how each retransmission protocol increased the reliability of communications, this analysis evaluated the system without using either retransmission protocol (Case-0), both Micro- and Macro-retransmission individually (Micro and Macro), and Micro- and Macro-retransmission combined. This thesis used a multimodal modeling methodology to prove that a performance and reliability analysis of WAHREN was possible, and provided the results of the analysis. A multimodal approach was necessary because these processes were driven by different mathematical models. The results from this analysis can be used as a framework for making design decisions for the Auger North communication system.

A NATURALISTIC COMPUTATIONAL MODEL OF HUMAN BEHAVIOR IN NAVIGATION AND SEARCH TASKS

Planning, navigation, and search are fundamental human cognitive abilities central to spatial problem solving in search and rescue, law enforcement, and military operations. Despite a wealth of literature concerning naturalistic spatial problem solving in animals, literature on naturalistic spatial problem solving in humans is comparatively lacking and generally conducted by separate camps among which there is little crosstalk. Addressing this deficiency will allow us to predict spatial decision making in operational environments, and understand the factors leading to those decisions. The present dissertation is comprised of two related efforts, (1) a set of empirical research studies intended to identify characteristics of planning, execution, and memory in naturalistic spatial problem solving tasks, and (2) a computational modeling effort to develop a model of naturalistic spatial problem solving. The results of the behavioral studies indicate that problem space hierarchical representations are linear in shape, and that human solutions are produced according to multiple optimization criteria. The Mixed Criteria Model presented in this dissertation accounts for global and local human performance in a traditional and naturalistic Traveling Salesman Problem. The results of the empirical and modeling efforts hold implications for basic and applied science in domains such as problem solving, operations research, human-computer interaction, and artificial intelligence.

A PHENOMENOLOGY OF MIMETIC LEARNING AND MULTIMODAL COGNITION: INTEGRATING EXPERIENTIAL KNOWLEDGE INTO PROGRAMS IN RHETORIC, COMPOSITION, AND TECHNICAL COMMUNICATION

My dissertation emphasizes a cognitive account of multimodality that explicitly integrates experiential knowledge work into the rhetorical pedagogy that informs so many composition and technical communication programs. In these disciplines, multimodality is widely conceived in terms of what Gunther Kress calls “socialsemiotic” modes of communication shaped primarily by culture. In the cognitive and neurolinguistic theories of Vittorio Gallese and George Lakoff, however, multimodality is described as a key characteristic of our bodies’ sensory-motor systems which link perception to action and action to meaning, grounding all communicative acts in knowledge shaped through body-engaged experience. I argue that this “situated” account of cognition – which closely approximates Maurice Merleau-Ponty’s phenomenology of perception, a major framework for my study – has pedagogical precedence in the mimetic pedagogy that informed ancient Sophistic rhetorical training, and I reveal that training’s multimodal dimensions through a phenomenological exegesis of the concept mimesis. Plato’s denigration of the mimetic tradition and his elevation of conceptual contemplation through reason, out of which developed the classic Cartesian separation of mind from body, resulted in a general degradation of experiential knowledge in Western education. But with the recent introduction into college classrooms of digital technologies and multimedia communication tools, renewed emphasis is being placed on the “hands-on” nature of inventive and productive praxis, necessitating a revision of methods of instruction and assessment that have traditionally privileged the acquisition of conceptual over experiential knowledge. The model of multimodality I construct from Merleau-Ponty’s phenomenology, ancient Sophistic rhetorical pedagogy, and current neuroscientific accounts of situated cognition insists on recognizing the significant role knowledges we acquire experientially play in our reading and writing, speaking and listening, discerning and designing practices.

Assessing Decision Making Skill in Complex and Dynamic Environments Using Representative and Simulated Tasks

Traditional decision making research has often focused on one's ability to choose from a set of prefixed options, ignoring the process by which decision makers generate courses of action (i.e., options) in-situ (Klein, 1993). In complex and dynamic domains, this option generation process is particularly critical to understanding how successful decisions are made (Zsambok & Klein, 1997). When generating response options for oneself to pursue (i.e., during the intervention-phase of decision making) previous research has supported quick and intuitive heuristics, such as the Take-The-First heuristic (TTF; Johnson & Raab, 2003). When generating predictive options for others in the environment (i.e., during the assessment-phase of decision making), previous research has supported the situational-model-building process described by Long Term Working Memory theory (LTWM; see Ward, Ericsson, & Williams, 2013). In the first three experiments, the claims of TTF and LTWM are tested during assessment- and intervention-phase tasks in soccer. To test what other environmental constraints may dictate the use of these cognitive mechanisms, the claims of these models are also tested in the presence and absence of time pressure. In addition to understanding the option generation process, it is important that researchers in complex and dynamic domains also develop tools that can be used by `real-world' professionals. For this reason, three more experiments were conducted to evaluate the effectiveness of a new online assessment of perceptual-cognitive skill in soccer. This test differentiated between skill groups and predicted performance on a previously established test and predicted option generation behavior. The test also outperformed domain-general cognitive tests, but not a domain-specific knowledge test when predicting skill group membership. Implications for theory and training, and future directions for the development of applied tools are discussed.