Using constructed-response and multiple-choice questions in undergraduate examinations

This paper looks at the use of constructed-response and multiple-choice questions in first year management examinations involving 2364 students over a three semester period in 2008-9. It also compares student’s performance in research and analytical assignments with their results in Multiple-Choice tests. The results show that students who perform poorly in non-multiple-choice tests are slightly advantaged by multiple-choice tests. However, students who display high distinction scores in other pieces of assessment do not receive any advantage from multiple-choice testing. Interestingly, the research also highlighted that female students perform better in their articulation of management techniques overall, but do not gain the same comparative advantage from multiple choice testing. The analysis of the assessment methods was extremely useful in predicting the level of failures for the unit in the subsequent year. Finding the right balance of assessment tasks in first year university business courses is extremely difficult, using multiple-choice tests can be useful tool as part of the assessment mix in management units.

Constructed Response Answers in a Multiple-Choice Universe

A scheme is introduced which allows computer readable multiple choice forms used in traditional examinations to be employed for constructed response items.

Kemian ylioppilaskokeen tehtävät summatiivisen arvioinnin välineenä

The main purpose of the research was to illustrate chemistry matriculation examination questions as a summative assessment tool, and represent how the questions have evolved over the years. Summative assessment and its various test item classifications, Finnish goal-oriented curriculum model, and Bloom’s Revised Taxonomy of Cognitive Objectives formed the theoretical framework for the research. The research data consisted of 257 chemistry questions from 28 matriculation examinations between 1996 and 2009. The analysed test questions were formulated according to the national upper secondary school chemistry curricula 1994, and 2003. Qualitative approach and theory-driven content analysis method were employed in the research. Peer review was used to guarantee the reliability of the results. The research was guided by the following questions: (a) What kinds of test item formats are used in chemistry matriculation examinations? (b) How the fundamentals of chemistry are included in the chemistry matriculation examination questions? (c) What kinds of cognitive knowledge and skills do the chemistry matriculation examination questions require? The research indicates that summative assessment was used diversely in chemistry matriculation examinations. The tests included various test item formats, and their combinations. The majority of the test questions were constructed-response items that were either verbal, quantitative, or experimental questions, symbol questions, or combinations of the aforementioned. The studied chemistry matriculation examinations seldom included selected-response items that can be either multiple-choice, alternate choice, or matching items. The relative emphasis of the test item formats differed slightly depending on whether the test was a part of an extensive general studies battery of tests in sciences and humanities, or a subject-specific test. The classification framework developed in the research can be applied in chemistry and science education, and also in educational research. Chemistry matriculation examinations are based on the goal-oriented curriculum model, and cover relatively well the fundamentals of chemistry included in the national curriculum. Most of the test questions related to the symbolism of chemical equation, inorganic and organic reaction types and applications, the bonding and spatial structure in organic compounds, and stoichiometry problems. Only a few questions related to electrolysis, polymers, or buffer solutions. None of the test questions related to composites. There were not any significant differences in the emphasis between the tests formulated according to the national curriculum 1994 or 2003. Chemistry matriculation examinations are cognitively demanding. The research shows that the majority of the test questions require higher-order cognitive skills. Most of the questions required analysis of procedural knowledge. The questions that only required remembering or processing metacognitive knowledge, were not included in the research data. The required knowledge and skill level varied slightly between the test questions in the extensive general studies battery of tests in sciences and humanities, and subject-specific tests administered since 2006. The proportion of the Finnish chemistry matriculation examination questions requiring higher-order cognitive knowledge and skills is very large compared to what is discussed in the research literature.

Assessment method difference : comparisons between international and domestic students within a first year undergraduate management course

This paper investigates the assessment methods and student results within a first year undergraduate management course offered within the business faculty of an Australian university. This course is compulsory for those studying for a commerce or management degree. The assessment results of full fee paying international students were compared with those of domestic students, during four teaching semesters in 2009 and 2010. Analysis compares 2,682 students’ numerical results for two constructed response assignments to their results for an examination comprising both multiple choice questions and constructed response questions. It also compares the results of international and domestic students across metropolitan, regional and rural campuses. However due to little comparison data for multiple campuses, findings are consolidated by domestic and international students, university-wide. International students were found to achieve lower results than domestic students for constructed response assessment tasks, but higher results than domestic students for multiple choice question assessments. These findings have implications for instructors eager to provide a level assessment playing field for both domestic and international students, enabling both groupings to take advantage of existing strengths but also to improve their weaknesses. This research led to a restructuring and rescheduling of assessment tasks for the 2012 academic year.

Avaliação do efeito de um programa de ensino de leitura e escrita sobre a fala de crianças usuárias de implante coclear

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Equivalência monetária em crianças surdas

No ensino de habilidades monetárias os procedimentos de escolha com o modelo (MTS) e de escolha com o modelo com resposta construída (CRMTS) têm se mostrado efetivos. Entretanto, há uma controvérsia sobre a importância dos pré-requisitos. O experimento 1 teve por objetivo verificar o efeito de um ensino de MTS e CRMTS sobre a equivalência monetária em crianças surdas com diferentes repertórios matemáticos. Participaram 10 crianças matriculadas em uma Unidade de Ensino Especializada distribuídas em dois grupos experimentais: Crianças com maior repertório matemático (Grupo I) e crianças com menor repertório matemático (Grupo II). As sessões foram realizadas na referida instituição. Um microcomputador foi usado com um software (REL 5.3.3 for Windows). Inicialmente, os participantes foram submetidos a pré-testes e a um pré-treino de identidade. Em seguida, ao ensino de relações condicionais via MTS entre valores monetários em LIBRAS e numerais decimais (AB), valor monetário em LIBRAS e figuras de moedas (AC) e a valor monetário em LIBRAS e figuras de notas (AD), seguidos dos testes de simetria e transitividade. Nos dois grupos, houve emergência de relações para a maioria dos participantes. Posteriormente, houve ensino via CRMTS de figuras de notas e numerais decimais (DB‟), seguido de testes de simetria e transitividade. Nos testes houve variabilidade no responder em ambos os grupos. Pós-testes e testes de manutenção mostraram desempenho consistente com o treino. O experimento 2 teve por objetivo replicar os resultados do experimento 1 verificando se modificações no ensino produziria equivalência monetária. Participaram 3 crianças surdas. O ambiente experimental, materiais e equipamentos foram os mesmos. O procedimento foi similar ao procedimento utilizado no experimento 1, mas com a introdução de algumas fases experimentais: 1) tentativas randomizadas de ensino, ensino de componentes numéricos e pré-treino de CRMTS. Os resultados indicam emergência de relações para os três participantes em MTS e emergência de poucas relações em CRMTS. Em ambos os experimentos a complexidade da tarefa foi uma das possíveis explicações do desempenho e os pré-requisitos não interferiram na a aprendizagem deste tipo de tarefa.

Accuracy in dictation after improvement of reading and copying skills in a student with learning difficulties

The aim of this study was to use systematic teaching in a clinical setting using software to teach reading and writing in one boy with learning difficulties and obtain accurate performance in dictation. In the pre-test, the student showed good performance in matching-to-sample tasks (96% in matching printed words to dictated words; 88% in matching pictures to dictated words), although he obtained a low percentage of correct answers in the dictation of constructed responses (52%) and manuscripts (24%). The teaching strengthened the selection of printed words matched to dictated words and copying words. The student obtained 100% correct answers in the teaching tasks. In post-tests of dictation, he obtained 100% correct answers in constructed response and 96% correct answers in manuscripts. The results indicate that carefully teaching copying may promote accuracy in the performance of responding to minimal units in dictation tasks.

Improving sixth grade student knowledge of ecology using fish rearing and release as a real-world context

This research project measured the effects of real-world content in a science classroom by determining change (deep knowledge of life science content, including ecosystems from MDE – Grade Level Content Expectations) in a subset of students (6th Grade Science) that may result from the addition of curriculum (real-world content of rearing trout in the classroom). Data showed large gains from the pre-test to post-test in students from both the experimental and control groups. The ecology unit with the implementation of real-world content [trout] was even more successful, and improved students’ deep knowledge of ecosystem content from Michigan’s Department of Education Grade Level Content Expectations. The gains by the experimental group on the constructed response section of the test, which included higher cognitive level items, were significant. Clinical interviews after the post-test confirmed increases in deep knowledge of ecosystem concepts in the experimental group, by revealing that a sample of experimental group students had a better grasp of important ecology concepts as compared to a sample of control group students.

Validation of the Item-Attribute Matrix in TIMSS-Mathematics Using Multiple Regression and the LSDM

The purposes of this study were (1) to validate of the item-attribute matrix using two levels of attributes (Level 1 attributes and Level 2 sub-attributes), and (2) through retrofitting the diagnostic models to the mathematics test of the Trends in International Mathematics and Science Study (TIMSS), to evaluate the construct validity of TIMSS mathematics assessment by comparing the results of two assessment booklets. Item data were extracted from Booklets 2 and 3 for the 8th grade in TIMSS 2007, which included a total of 49 mathematics items and every student's response to every item. The study developed three categories of attributes at two levels: content, cognitive process (TIMSS or new), and comprehensive cognitive process (or IT) based on the TIMSS assessment framework, cognitive procedures, and item type. At level one, there were 4 content attributes (number, algebra, geometry, and data and chance), 3 TIMSS process attributes (knowing, applying, and reasoning), and 4 new process attributes (identifying, computing, judging, and reasoning). At level two, the level 1 attributes were further divided into 32 sub-attributes. There was only one level of IT attributes (multiple steps/responses, complexity, and constructed-response). Twelve Q-matrices (4 originally specified, 4 random, and 4 revised) were investigated with eleven Q-matrix models (QM1 ~ QM11) using multiple regression and the least squares distance method (LSDM). Comprehensive analyses indicated that the proposed Q-matrices explained most of the variance in item difficulty (i.e., 64% to 81%). The cognitive process attributes contributed to the item difficulties more than the content attributes, and the IT attributes contributed much more than both the content and process attributes. The new retrofitted process attributes explained the items better than the TIMSS process attributes. Results generated from the level 1 attributes and the level 2 attributes were consistent. Most attributes could be used to recover students' performance, but some attributes' probabilities showed unreasonable patterns. The analysis approaches could not demonstrate if the same construct validity was supported across booklets. The proposed attributes and Q-matrices explained the items of Booklet 2 better than the items of Booklet 3. The specified Q-matrices explained the items better than the random Q-matrices.

Institutional response and Indigenous experiences of Cyclone Tracy

Early on Christmas morning 1974 Tropical Cyclone Tracy, a Category 4 storm, devastated the Northern Territory city of Darwin leaving only 6% of the city’s housing habitable. The extent of the disaster was largely the result of unregulated and poorly constructed buildings, predominantly housing. While the engineering and reconstruction process demonstrated a very successful response and adaptation to an existing and future risk, the impact of the cyclone of the local community and its Indigenous population in particular, had not been well recorded. NCCARF therefore commissioned a report on the Indigenous experience of Cyclone Tracy to document how Indigenous people were impacted by, responded to, and recovered from Cyclone Tracy in comparison to non-Indigenous groups. The report also considers the research literature on disasters and Indigenous people in the Northern Territory, with a specific focus on cyclones, and considers the socio-political context of Indigenous communities in Darwin prior to Cyclone Tracy.

Road operation phase sustainability indicator as a response to the climate change phenomena

Road infrastructure is a major contributor of greenhouse gas (GHG) around the world. Once constructed, a road becomes a part of a road network and is subjected to recurrent maintenance/rehabilitation activities. Studies to date are mostly aimed at the development of sustainability indicators that deal with the material and construction phases of a road when it is constructed. The operation phase is infrequently studied and there is a need for sustainability indicators to be developed relating to this phase to better understand the GHG emissions as a proper response to the climate change phenomena. During the operation phase, maintenance/rehabilitation activities are undertaken based on certain agreed intervention criteria that do not include environmental implications relating to the climate change aspect properly. Availability of appropriate indicators may, therefore, assist in sustainable road asset maintenance management. This paper presents the findings of a literature based study and has proposed a way forward to develop a key “road operation phase” environmental indicator, which can contribute to road operation phase carbon footprint management based on a comprehensive road life cycle system boundary model. The proposed indicator can address multiple aspects of high impact road operation life environmental components such as: pavement rolling resistance, albedo, material, traffic congestion and lighting, based on availability of relevant scientific knowledge. Development of the indicator to appropriate level would offset the impacts of these components significantly and contribute to sustainable road operation management.