Automatized Generation of Metadata and Calibrated Test Item Banks for E-learning

Report published in the Proceedings of the National Conference on "Education in the Information Society", Plovdiv, May, 2013

Numeracy test item readability during transition from pre-school to school

A key test used in Australia to assess the mathematical knowledge of young children uses illustrations of objects such as coins and three-dimensional shapes. This study explored the effects of giving 104 kindergarten children, aged 4-5 years, the questions with either moveable objects or illustrations. It
was found that children who were categorized by their teachers as having “higher levels of numeracy” scored well on test questions using either illustrations or objects, while children who were categorized as having “lower levels of numeracy” scored higher with objects than with illustrations. This result could have implications for consideration of test item readability in relation to graphicacy.

Item and task difficulty in a B2 reading examination: perceptions of test-takers and CEFR alignment experts compared with psychometric measurements

The article presents a study of a CEFR B2-level reading subtest that is part of the Slovenian national secondary school leaving examination in English as a foreign language, and compares the test-taker actual performance (objective difficulty) with the test-taker and expert perceptions of item difficulty (subjective difficulty). The study also analyses the test-takers’ comments on item difficulty obtained from a while-reading questionnaire. The results are discussed in the framework of the existing research in the fields of (the assessment of) reading comprehension, and are addressed with regard to their implications for item-writing, FL teaching and curriculum development.

Kemian ylioppilaskokeen tehtävät summatiivisen arvioinnin välineenä

The main purpose of the research was to illustrate chemistry matriculation examination questions as a summative assessment tool, and represent how the questions have evolved over the years. Summative assessment and its various test item classifications, Finnish goal-oriented curriculum model, and Bloom’s Revised Taxonomy of Cognitive Objectives formed the theoretical framework for the research. The research data consisted of 257 chemistry questions from 28 matriculation examinations between 1996 and 2009. The analysed test questions were formulated according to the national upper secondary school chemistry curricula 1994, and 2003. Qualitative approach and theory-driven content analysis method were employed in the research. Peer review was used to guarantee the reliability of the results. The research was guided by the following questions: (a) What kinds of test item formats are used in chemistry matriculation examinations? (b) How the fundamentals of chemistry are included in the chemistry matriculation examination questions? (c) What kinds of cognitive knowledge and skills do the chemistry matriculation examination questions require? The research indicates that summative assessment was used diversely in chemistry matriculation examinations. The tests included various test item formats, and their combinations. The majority of the test questions were constructed-response items that were either verbal, quantitative, or experimental questions, symbol questions, or combinations of the aforementioned. The studied chemistry matriculation examinations seldom included selected-response items that can be either multiple-choice, alternate choice, or matching items. The relative emphasis of the test item formats differed slightly depending on whether the test was a part of an extensive general studies battery of tests in sciences and humanities, or a subject-specific test. The classification framework developed in the research can be applied in chemistry and science education, and also in educational research. Chemistry matriculation examinations are based on the goal-oriented curriculum model, and cover relatively well the fundamentals of chemistry included in the national curriculum. Most of the test questions related to the symbolism of chemical equation, inorganic and organic reaction types and applications, the bonding and spatial structure in organic compounds, and stoichiometry problems. Only a few questions related to electrolysis, polymers, or buffer solutions. None of the test questions related to composites. There were not any significant differences in the emphasis between the tests formulated according to the national curriculum 1994 or 2003. Chemistry matriculation examinations are cognitively demanding. The research shows that the majority of the test questions require higher-order cognitive skills. Most of the questions required analysis of procedural knowledge. The questions that only required remembering or processing metacognitive knowledge, were not included in the research data. The required knowledge and skill level varied slightly between the test questions in the extensive general studies battery of tests in sciences and humanities, and subject-specific tests administered since 2006. The proportion of the Finnish chemistry matriculation examination questions requiring higher-order cognitive knowledge and skills is very large compared to what is discussed in the research literature.

Instructional strategies integrating cognitive style construct : a meta-knowledge processing model

The overarching goal of this dissertation was to evaluate the contextual components of instructional strategies for the acquisition of complex programming concepts. A meta-knowledge processing model is proposed, on the basis of the research findings, thereby facilitating the selection of media treatment for electronic courseware. When implemented, this model extends the work of Smith (1998), as a front-end methodology, for his glass-box interpreter called Bradman, for teaching novice programmers. Technology now provides the means to produce individualized instructional packages with relative ease. Multimedia and Web courseware development accentuate a highly graphical (or visual) approach to instructional formats. Typically, little consideration is given to the effectiveness of screen-based visual stimuli, and curiously, students are expected to be visually literate, despite the complexity of human-computer interaction. Visual literacy is much harder for some people to acquire than for others! (see Chapter Four: Conditions-of-the-Learner) An innovative research programme was devised to investigate the interactive effect of instructional strategies, enhanced with text-plus-textual metaphors or text-plus-graphical metaphors, and cognitive style, on the acquisition of a special category of abstract (process) programming concept. This type of concept was chosen to focus on the role of analogic knowledge involved in computer programming. The results are discussed within the context of the internal/external exchange process, drawing on Ritchey's (1980) concepts of within-item and between-item encoding elaborations. The methodology developed for the doctoral project integrates earlier research knowledge in a novel, interdisciplinary, conceptual framework, including: from instructional science in the USA, for the concept learning models; British cognitive psychology and human memory research, for defining the cognitive style construct; and Australian educational research, to provide the measurement tools for instructional outcomes. The experimental design consisted of a screening test to determine cognitive style, a pretest to determine prior domain knowledge in abstract programming knowledge elements, the instruction period, and a post-test to measure improved performance. This research design provides a three-level discovery process to articulate: 1) the fusion of strategic knowledge required by the novice learner for dealing with contexts within instructional strategies 2) acquisition of knowledge using measurable instructional outcome and learner characteristics 3) knowledge of the innate environmental factors which influence the instructional outcomes This research has successfully identified the interactive effect of instructional strategy, within an individual's cognitive style construct, in their acquisition of complex programming concepts. However, the significance of the three-level discovery process lies in the scope of the methodology to inform the design of a meta-knowledge processing model for instructional science. Firstly, the British cognitive style testing procedure, is a low cost, user friendly, computer application that effectively measures an individual's position on the two cognitive style continua (Riding & Cheema,1991). Secondly, the QUEST Interactive Test Analysis System (Izard,1995), allows for a probabilistic determination of an individual's knowledge level, relative to other participants, and relative to test-item difficulties. Test-items can be related to skill levels, and consequently, can be used by instructional scientists to measure knowledge acquisition. Finally, an Effect Size Analysis (Cohen,1977) allows for a direct comparison between treatment groups, giving a statistical measurement of how large an effect the independent variables have on the dependent outcomes. Combined with QUEST's hierarchical positioning of participants, this tool can assist in identifying preferred learning conditions for the evaluation of treatment groups. By combining these three assessment analysis tools into instructional research, a computerized learning shell, customised for individuals' cognitive constructs can be created (McKay & Garner,1999). While this approach has widespread application, individual researchers/trainers would nonetheless, need to validate with an extensive pilot study programme (McKay,1999a; McKay,1999b), the interactive effects within their specific learning domain. Furthermore, the instructional material does not need to be limited to a textual/graphical comparison, but could be applied to any two or more instructional treatments of any kind. For instance: a structured versus exploratory strategy. The possibilities and combinations are believed to be endless, provided the focus is maintained on linking of the front-end identification of cognitive style with an improved performance outcome. My in-depth analysis provides a better understanding of the interactive effects of the cognitive style construct and instructional format on the acquisition of abstract concepts, involving spatial relations and logical reasoning. In providing the basis for a meta-knowledge processing model, this research is expected to be of interest to educators, cognitive psychologists, communications engineers and computer scientists specialising in computer-human interactions.

A construção e validação de um teste com referência a critério para medir as competências básicas de leitura e escrita para o Programa de Alfabetização Funcional do MOBRAL

A necessidade de se desenvolverem instrumentos adequados a avaliação de programas educacionais é premente no momento atual. Segundo os estudos mais recentes no campo da medida educacional, os testes com referência a critério são os que atendem a esta necessidade. Este estudo teve por objetivo a construção de dois testes deste tipo para a avaliação do Programa de Alfabetização Funcional do MOBRAL nos dominios da aquisição das competências basicas em leitura e escrita. Inicialmente foram selecionadas e descritas quatro competências para serem medidas por cada teste. As competências foram selecionadas tendo como base os objetivos do programa apresentados no Roteiro de Orientação ao Alfabetizador (1977). A segunda etapa do estudo consistiu na criação das especificações para cada competência selecionadas nos dois dominios. A partir destas especificações foram construidos 32 itens para o teste de leitura e 40 para o de escrita. Estes itens foram, distribuidos randomicamente em 5 formas experimentais para cada teste. O teste de leitura é composto de itens de múltipla escolha e o teste de escrita de itens de resposta construida. Os criterios de correção e os padrões minimos de desempenho foram estabelecidos para cada teste. Para a analise destes padrões minimos foram calculados as medias e os desvios-padrão para cada forma aplicada. Estes resultados permitem a manutenção dos padrões minimos usados em estudos posteriores. A análise dos itens foi feita em duas etapas: na primeira foram calculados os indices de discriminância pelo processo tradicional e na segunda os indices de sensitivida de pelo processo desenvolvido por Gronlund (1976). Os resultados obtidos permitiram o aproveitamento de todos os itens. Os itens foram divididos em duas formas equivalentes para cada teste, cada um pode ser subdividido em três subtestes para a medida isolada das competências descritas. Para o exame de validade foram estabelecidos os indices de validade descritiva e de seleção de dominio. A fidedignidade dos testes ficou assegurada com a construção de mais de 5 itens para competência Gronlund (1976).

Reação postural de pacientes com doença de Parkinson e idosos sadios no teste de retropulsão instrumentado

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)

The cognitive reflection test (CRT) is a short measure of a person's ability to resist intuitive response tendencies and to produce a normatively correct response, which is based on effortful reasoning. Although the CRT is a very popular measure, its psychometric properties have not been extensively investigated. A major limitation of the CRT is the difficulty of the items, which can lead to floor effects in populations other than highly educated adults. The present study aimed at investigating the psychometric properties of the CRT applying item response theory analyses (a two-parameter logistic model) and at developing a new version of the scale (the CRT-long), which is appropriate for participants with both lower and higher levels of cognitive reflection. The results demonstrated the good psychometric properties of the original, as well as the new scale. The validity of the new scale was also assessed by measuring correlations with various indicators of intelligence, numeracy, reasoning and decision-making skills, and thinking dispositions. Moreover, we present evidence for the suitability of the new scale to be used with developmental samples. Finally, by comparing the performance of adolescents and young adults on the CRT and CRT-long, we report the first investigation into the development of cognitive reflection.

The Use of the Rey 15-Item Test and Recognition Trial to Evaluate Suboptimal Effort in Neuropsychological Assessment with a Pediatric Mild TBI Sample

The utilization of symptom validity tests (SVTs) in pediatric assessment is receiving increasing empirical support. The Rey 15-Item Test (FIT) is an SVT commonly used in adult assessment, with limited research in pediatric populations. Given that FIT classification statistics across studies to date have been quite variable, Boone, Salazar, Lu, Warner-Chacon, and Razani (2002) developed a recognition trial to use with the original measure to enhance accuracy. The current study aims to assess the utility of the FIT and recognition trial in a pediatric mild traumatic brain injury (TBI) sample (N = 112; M = 14.6 years), in which a suboptimal effort base rate of 17% has been previously established (Kirkwood & Kirk, 2010). All participants were administered the FIT as part of an abbreviated neuropsychological evaluation; failure on the Medical Symptom Validity Test (MSVT) was used as the criterion for suspect effort. The traditional adult cut-off score of(99%), but poor sensitivity (6%). When the recognition trial was also utilized, a combination score of(sensitivity = 64%, specificity = 93%). Results indicate that the FIT with recognition trial may be useful in the assessment of pediatric suboptimal effort, at least among relatively high functioning children following mild TBI.

Hand receipt covering contents of components of end item (COEI), basic issue items (BII), and additional authorization list (AAL) for radio test set AN/PRM-34 (NSN 6625-01-094-5646).

"2 September 1983."