Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

Individual differences in the processing of punishment and reward cues : an application to road safety messages

This thesis examined the extent to which individual differences, as conceptualised by the revised Reinforcement Sensitivity Theory, influenced young drivers' information processing and subsequent acceptance of anti-speeding messages. Using a multi-method approach, the findings highlighted the utility of combining objective measures (a cognitive response time task and electroencephalography) with self-report measures to assess message processing and message acceptance, respectively. This body of research indicated that responses to anti-speeding messages may differ depending on an individual's personality disposition. Overall, the research provided further insight into the development of message strategies to target high risk drivers.

Estimating second language productive vocabulary size: A Capture-Recapture approach

This study provides validity evidence for the Capture-Recapture (CR) method, borrowed from ecology, as a measure of second language (L2) productive vocabulary size (PVS). Two separate “captures” of productive vocabulary were taken using written word association tasks (WAT). At Time 1, 47 bilinguals provided at least 4 associates to each of 30 high-frequency stimulus words in English, their first language (L1), and in French, their L2. A few days later (Time 2), this procedure was repeated with a different set of stimulus words in each language. Since the WAT was used, both Lex30 and CR PVS scores were calculated in each language. Participants also completed an animacy judgment task assessing the speed and efficiency of lexical access. Results indicated that, in both languages, CR and Lex30 scores were significantly positively correlated (evidence of convergent validity). CR scores were also significantly larger in the L1, and correlated significantly with the speed of lexical access in the L2 (evidence of construct validity). These results point to the validity of the technique for estimating relative L2 PVS. However, CR scores are not a direct indication of absolute vocabulary size. A discussion of the method’s underlying assumptions and their implications for interpretation are provided.

A meta-analysis of the correlation between heritage language and ethnic identity

The interaction between heritage language (HL) and ethnic identity has gained increasing scholarly attention over the past decades. Numerous quantitative studies have investigated and vindicated this interaction within certain contexts. Nevertheless, quantitative evidence on this interaction across contexts is absent to date. The current meta-analysis aims to make a contribution in this regard. By integrating relevant studies, this meta-analysis presents a powerful estimation of the reality in relation to the interaction between HL and ethnic identity. By virtue of certain retrieval strategies and selection criteria, the meta-analysis includes 43 data-sets emerging from 18 studies that have addressed the statistical correlation between the proficiency of HL and the sense of ethnic identity associated with different ethnic groups. When contrasted to one another, the results of these included studies are significantly different. However, when combined together, these studies point to a statistically significant moderate positive correlation between sense of ethnic identity and proficiency of HL across different ethnic groups. This result has a medium effect. The meta-analysis also inspires some methodological and theoretical discussions.

Devenir plurilingue : la perspective des littératies multiples

Abstract - English Multiple literacies refers to reading, reading the world and self. This article proposes an understanding of reading that goes beyond its definition in psychology and applied linguistics. This longitudinal project is interested in a conceptualisation of what reading is, how it functions and what it produces in becoming multilingual. Reading is explored through the lens of an empirical study involving five female pupils from senior Kindergarten to Grade 3 observed and interviewed in relation to activities at school and at home. The study took place in Ottawa schools where French is the sole language of instruction. Reading in the context of multiple literacies is conceptualised to disrupt /deterritorialise and to be immanent, offering the potentiality to go beyond what is to what could be. Becoming multilingual is a continuous movement involving networks of rhizomatic connections and reading the world and self. Résumé - Francais Les littératies multiples se réfèrent à la lecture, la lecture du monde et la lecture de soi. Cet article propose une compréhension de la lecture qui dépasse sa définition usuelle en psychologie et en linguistique appliquée. Ce projet longitudinal porte sur la conceptualisation de la lecture, son fonctionnement et ce qu’elle produit dans le devenir plurilingue. La lecture est examinée selon l’optique d’une étude empirique durant laquelle cinq écolières du jardin d’enfants à la 3e année étaient observées et interviewées par rapport à des activités à l’école et à la maison. L’étude a eu lieu dans des écoles d’Ottawa dont la seule langue d’enseignement est le français. Dans le contexte des littératies multiples, la lecture est conceptualisée comme étant perturbatrice/déterritorialisante et immanente. Elle offre la potentialité d’aller au-delà de ce qui est vers ce qui pourrait être. Devenir plurilingue est un mouvement continu faisant appel à des réseaux de connexions rhizomatiques et à la lecture du monde et de soi.

Encouraging Empathy through Picture Books about Migration

Australia has become one of the most highly multilingual and multicultural societies in the world today with people descending from 270 ancestries, who speak more than 260 languages (Commonwealth of Australia, 2011). Immigration is something that children encounter in their daily lives either through personal experience or through witnessing the lives of migrants at school, in the community, or through popular media, including children’s literature. Schools are frequently the initial interface for individuals who resettle in Australia and they ‘play a significant role in establishing meaningful connections to Australian society and a sense of belonging in Australia’ (Uptin, Wright, & Harwood, 2013, p. 1). Children's literature about cultural and ethnic diversity explores the impacts of migration and related issues creating ‘imaginary realms’ (Dudek & Ommundsen, 2007). These fictional interpretations of the migrant experience or the experience of migration are supported by distinctive “real life” cultural experiences. Picture books furnish teachers and students with an accessible means to investigate these complex issues through sensitive discussions. This chapter investigates how picture books about migration help deepen children’s perceptive understanding of migrants’ plights, and thereby nurture tolerance and empathy.

Teachers-as-writers: Writing information texts in English and vernacular

Our contribution to this volume is not on the work of the teacher who inspires the child writer, but the teacher as the writer and illustrator of multilingual texts for classroom use that inspires the child reader. This chapter focuses on a first time teacher writer from Fiji, Bereta , who participated in a two day writing workshop known as the Information Text Awareness Project (hereafter ITAP). This chapter commences with an overview of the ITAP which was conducted in Nadi, Fiji, in 2012 with Bereta and 17 teachers from urban, semi-urban and rural contexts within the Nadi educational district. The politics of presenting Western ways of knowing to teachers from diverse cultural and linguistic contexts via a Western pedagogical approach is explored in the second section. We believe that this work involves a moral dimension that needs careful consideration. The third section outlines the eight stages of ITAP where teacher writers such as Bereta produced an English and a vernacular information text for use in their classrooms. The outline of the eight stages of ITAP is justified with links to the research literature. The final section recounts Bereta’s interview data where she talks about using the newly created English and vernacular information texts in the classroom and the community’s response to her inaugural publications. The findings may be of interest to those seeking to establish an adult writing cooperative to produce English and vernacular information texts for classroom use.

Forward: On Chineseness

A critical commentary on issues of Chinese identity, habitus and language.

Becoming bilingual: Children’s insights about making friends in bilingual settings

The majority of the world speaks more than one language yet the impact of learning a second language has rarely been studied from a child’s perspective. This paper describes monolingual children’s insights into becoming bilingual at four time points: two months before moving to another country (while living in Australia), as well as one, six, and twelve months after moving to Germany. The participants were two monolingual English-speaking siblings (a male aged 7- to 8-years and a female aged 9- to 10-years) who subsequently learned to speak German. At each of the four time points, interviews were undertaken with each child using child-friendly drawing and questionnaire techniques. Three themes were identified: (1) the children’s awareness of language competence; (2) inclusion factors, and; (3) exclusion factors that influenced friendship formation. The impact of language ability on making friends was a dominant theme that arose across the four time points and was triangulated across data collection methods. The children made friends with others who had similar language competence in German, even though they were younger, and did not share the same first language. Age-matched peers who were more competent in German were less likely to be described as friends. Across all three themes, the playground was highlighted by both children as the key site where becoming bilingual most strongly impacted initiation and negotiation of friendships. Becoming bilingual impacted the children’s friendship formation and socialization opportunities with more competent language users.

Probabilistic orthographic cues to grammatical category in the brain

What helps us determine whether a word is a noun or a verb, without conscious awareness? We report on cues in the way individual English words are spelled, and, for the first time, identify their neural correlates via functional magnetic resonance imaging (fMRI). We used a lexical decision task with trisyllabic nouns and verbs containing orthographic cues that are either consistent or inconsistent with the spelling patterns of words from that grammatical category. Significant linear increases in response times and error rates were observed as orthography became less consistent, paralleled by significant linear decreases in blood oxygen level dependent (BOLD) signal in the left supramarginal gyrus of the left inferior parietal lobule, a brain region implicated in visual word recognition. A similar pattern was observed in the left superior parietal lobule. These findings align with an emergentist view of grammatical category processing which results from sensitivity to multiple probabilistic cues.

Neural correlates of semantic priming for ambiguous words: An event-related fMRI study

We investigated the neural correlates of semantic priming by using event-related fMRI to record blood oxygen level dependent (BOLD) responses while participants performed speeded lexical decisions (word/nonword) on visually presented related versus unrelated prime-target pairs. A long stimulus onset asynchrony of 1000 ms was employed, which allowed for increased controlled processing and selective frequency-based ambiguity priming. Conditions included an ambiguous word prime (e.g. bank) and a target related to its dominant (e.g. money) or subordinate meaning (e.g. river). Compared to an unrelated condition, primed dominant targets were associated with increased activity in the LIFG, the right anterior cingulate and superior temporal gyrus, suggesting postlexical semantic integrative mechanisms, while increased right supramarginal activity for the unrelated condition was consistent with expectancy based priming. Subordinate targets were not primed and were associated with reduced activity primarily in occipitotemporal regions associated with word recognition, which may be consistent with frequency-based meaning suppression. These findings provide new insights into the neural substrates of semantic priming and the functional-anatomic correlates of lexical ambiguity suppression mechanisms.

Brain activity during automatic semantic priming revealed by event-related functional magnetic resonance imaging

Semantic priming occurs when a subject is faster in recognising a target word when it is preceded by a related word compared to an unrelated word. The effect is attributed to automatic or controlled processing mechanisms elicited by short or long interstimulus intervals (ISIs) between primes and targets. We employed event-related functional magnetic resonance imaging (fMRI) to investigate blood oxygen level dependent (BOLD) responses associated with automatic semantic priming using an experimental design identical to that used in standard behavioural priming tasks. Prime-target semantic strength was manipulated by using lexical ambiguity primes (e.g., bank) and target words related to dominant or subordinate meaning of the ambiguity. Subjects made speeded lexical decisions (word/nonword) on dominant related, subordinate related, and unrelated word pairs presented randomly with a short ISI. The major finding was a pattern of reduced activity in middle temporal and inferior prefrontal regions for dominant versus unrelated and subordinate versus unrelated comparisons, respectively. These findings are consistent with both a dual process model of semantic priming and recent repetition priming data that suggest that reductions in BOLD responses represent neural priming associated with automatic semantic activation and implicate the left middle temporal cortex and inferior prefrontal cortex in more automatic aspects of semantic processing.

Dopaminergic neuromodulation of semantic processing: A 4-T fMRI study with levodopa

There is emerging evidence that alterations in dopaminergic transmission can influence semantic processing, yet the neural mechanisms involved are unknown. The influence of levodopa (L-DOPA) on semantic priming was investigated in healthy individuals (n=20) using event-related functional magnetic resonance imaging with a randomized, double-blind crossover design. Critical prime-target pairs consisted of a lexical ambiguity prime and 1) a target related to the dominant meaning of the prime (e.g., bank-money), 2) a target related to the subordinate meaning (e.g., fence-sword), or 3) an unrelated target (e.g., ball-desk). Behavioral data showed that both dominant and subordinate meanings were primed on placebo. In contrast, there was preserved priming of dominant meanings and no significant priming of subordinate meanings on L-DOPA, the latter associated with decreased anterior cingulate and dorsal prefrontal cortex activity. Dominant meaning activation on L-DOPA was associated with increased activity in the left rolandic operculum and left middle temporal gyrus. These findings suggest that L-DOPA enhances frequency-based semantic focus via prefrontal and temporal modulation of automatic semantic priming and through engagement of anterior cingulate mechanisms supporting attentional/controlled priming.

A perfusion fMRI investigation of thematic and categorical context effects in the spoken production of object names

The context in which objects are presented influences the speed at which they are named. We employed the blocked cyclic naming paradigm and perfusion functional magnetic resonance imaging (fMRI) to investigate the mechanisms responsible for interference effects reported for thematicallyand categorically related compared to unrelated contexts. Naming objects in categorically homogeneous contexts induced a significant interference effect that accumulated from the second cycle onwards. This interference effect was associated with significant perfusion signal decreases in left middle and posterior lateral temporal cortex and the hippocampus. By contrast, thematically homogeneous contexts facilitated naming latencies significantly in the first cycle and did not differ from heterogeneous contexts thereafter, nor were they associated with any perfusion signal changes compared to heterogeneous contexts. These results are interpreted as being consistent with an account in which the interference effect both originates and has its locus at the lexical level, with an incremental learning mechanism adapting the activation levels of target lexical representations following access. We discuss the implications of these findings for accounts that assume thematic relations can be active lexical competitors or assume mandatory involvement of top-down control mechanisms in interference effects during naming.

Differential processing of thematic and categorical conceptual relations in spoken word production

Studies of semantic context effects in spoken word production have typically distinguished between categorical (or taxonomic) and associative relations. However, associates tend to confound semantic features or morphological representations, such as whole-part relations and compounds (e.g., BOAT-anchor, BEE-hive). Using a picture-word interference paradigm and functional magnetic resonance imaging (fMRI), we manipulated categorical (COW-rat) and thematic (COW-pasture) TARGET-distractor relations in a balanced design, finding interference and facilitation effects on naming latencies, respectively, as well as differential patterns of brain activation compared with an unrelated distractor condition. While both types of distractor relation activated the middle portion of the left middle temporal gyrus (MTG) consistent with retrieval of conceptual or lexical representations, categorical relations involved additional activation of posterior left MTG, consistent with retrieval of a lexical cohort. Thematic relations involved additional activation of the left angular gyrus. These results converge with recent lesion evidence implicating the left inferior parietal lobe in processing thematic relations and may indicate a potential role for this region during spoken word production.