Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

Innovative use of Blackboard® to assess laboratory skills

A novel application of the popular web instruction architecture Blackboard Academic Suite® is described. The method was applied to a large number of students to assess quantitatively the accuracy of each student’s laboratory skills. The method provided immediate feedback to students on their personal skill level, replaced labour-intensive scrutiny of laboratory skills by teaching staff and identified immediately those students requiring further individual assistance in mastering the skill under evaluation. The method can be used for both formative and summative assessment. When used formatively, the assessment can be repeated by the student without penalty until the skill is mastered. When used for summative assessment, the method can save the teacher much time and effort in assessing laboratory skills of vital importance to students in the real world.

Case study of collaborative learning in two contexts : what do students gain?

This book presents important research advances in the study of teaching and teacher research as well as a review of motivation in education; mentoring; an evaluation of online learning; educational change and computer-assisted teaching.

The design and development of an undergraduate signal processing laboratory

The School of Electrical and Electronic Systems Engineering of Queensland University of Technology (like many other universities around the world) has recognised the importance of complementing the teaching of signal processing with computer based experiments. A laboratory has been developed to provide a "hands-on" approach to the teaching of signal processing techniques. The motivation for the development of this laboratory was the cliche "What I hear I remember but what I do I understand." The laboratory has been named as the "Signal Computing and Real-time DSP Laboratory" and provides practical training to approximately 150 final year undergraduate students each year. The paper describes the novel features of the laboratory, techniques used in the laboratory based teaching, interesting aspects of the experiments that have been developed and student evaluation of the teaching techniques

Intercultural usability of language learning websites

Website usability can be defined as the ease of use of websites. General usability, pedagogical usability, technical usability and intercultural usability can be considered and examined for the understanding of the usability of language learning websites, which requires a discipline-specific approach. In the field of computer-assisted language learning, usability issues have been addressed mainly in terms of evaluation criteria and have been commonly discussed in relation to user expectations and user experiences. In spite of a growing interest in intercultural language learning, however, little research on intercultural usability of language learning websites has been published yet. There is a need to answer the question of how language learning websites integrate the target language and culture for the development of intercultural sensitivity and competence. This article explores intercultural aspects of language learning websites and presents usability guidelines for designing intercultural language learning websites.

Evaluation of Attacks and Countermeasures in Large Scale Networks

Computer worms represent a serious threat for modern communication infrastructures. These epidemics can cause great damage such as financial losses or interruption of critical services which support lives of citizens. These worms can spread with a speed which prevents instant human intervention. Therefore automatic detection and mitigation techniques need to be developed. However, if these techniques are not designed and intensively tested in realistic environments, they may cause even more harm as they heavily interfere with high volume communication flows. We present a simulation model which allows studies of worm spread and counter measures in large scale multi-AS topologies with millions of IP addresses.

A computer image analysis system for microvessel density measurement in solid tumours

Microvessel density (MVD) is a widely used surrogate measure of angiogenesis in pathological specimens and tumour models. Measurement of MVD can be achieved by several methods. Automation of counting methods aims to increase the speed, reliability and reproducibility of these techniques. The image analysis system described here enables MVD measurement to be carried out with minimal expense in any reasonably equipped pathology department or laboratory. It is demonstrated that the system translates easily between tumour types which are suitably stained with minimal calibration. The aim of this paper is to offer this technique to a wider field of researchers in angiogenesis.

Toxicity profile of the immunomodulatory thalidomide analogue, lenalidomide : Phase I clinical trial of three dosing schedules in patients with solid malignancies

Thalidomide is an anti-angiogenic agent currently used to treat patients with malignant cachexia or multiple myeloma. Lenalidomide (CC-5013) is an immunomodulatory thalidomide analogue licensed in the United States of America (USA) for the treatment of a subtype of myelodysplastic syndrome. This two-centre, open-label phase I study evaluated dose-limiting toxicities in 55 patients with malignant solid tumours refractory to standard chemotherapies. Lenalidomide capsules were consumed once daily for 12 weeks according to one of the following three schedules: (I) 25 mg daily for the first 7 d, the daily dose increased by 25 mg each week up to a maximum daily dose of 150 mg; (II) 25 mg daily for 21 d followed by a 7-d rest period, the 4-week cycle repeated for 3 cycles; (III) 10 mg daily continuously. Twenty-six patients completed the study period. Two patients experienced a grade 3 hypersensitivity rash. Four patients in cohort I and 4 patients in cohort II suffered grade 3 or 4 neutropaenia. In 2 patients with predisposing medical factors, grade 3 cardiac dysrhythmia was recorded. Grade 1 neurotoxicity was detected in 6 patients. One complete and two partial radiological responses were measured by computed tomography scanning; 8 patients had stable disease after 12 weeks of treatment. Fifteen patients remained on treatment as named patients; 1 with metastatic melanoma remains in clinical remission 3.5 years from trial entry. This study indicates the tolerability and potential clinical efficacy of lenalidomide in patients with advanced solid tumours who have previously received multi-modality treatment. Depending on the extent of myelosuppressive pre-treatment, dose schedules (II) or (III) are advocated for large-scale trials of long-term administration. © 2006 Elsevier Ltd. All rights reserved.

Making visible the coding process : Using qualitative data software in a post-structural study

Qualitative research methods require transparency to ensure the ‘trustworthiness’ of the data analysis. The intricate processes of organizing, coding and analyzing the data are often rendered invisible in the presentation of the research findings, which requires a ‘leap of faith’ for the reader. Computer assisted data analysis software can be used to make the research process more transparent, without sacrificing rich, interpretive analysis by the researcher. This article describes in detail how one software package was used in a poststructural study to link and code multiple forms of data to four research questions for fine-grained analysis. This description will be useful for researchers seeking to use qualitative data analysis software as an analytic tool.