Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

Automated melanoma diagnosis system

An automated melanoma diagnosis system, the so-called Skin Polar-probe, was developed to improve the chances of early detection of skin cancers and help save the lives of melanoma victims. The system will offer unique benefits to aid early detection of melanoma - the key to reducing deaths caused by this cancer.

Accelerating taxonomic discovery through automated character extraction

This paper discusses the following key messages. Taxonomy is (and taxonomists are) more important than ever in times of global change. Taxonomic endeavour is not occurring fast enough: in 250 years since the creation of the Linnean Systema Naturae, only about 20% of Earth's species have been named. We need fundamental changes to the taxonomic process and paradigm to increase taxonomic productivity by orders of magnitude. Currently, taxonomic productivity is limited principally by the rate at which we capture and manage morphological information to enable species discovery. Many recent (and welcomed) initiatives in managing and delivering biodiversity information and accelerating the taxonomic process do not address this bottleneck. Development of computational image analysis and feature extraction methods is a crucial missing capacity needed to enable taxonomists to overcome the taxonomic impediment in a meaningful time frame. Copyright © 2009 Magnolia Press.

Automated fit quantification of tibial nail designs during the insertion using computer three-dimensional modelling

Intramedullary nailing is the standard fixation method for displaced diaphyseal fractures of the tibia. An optimal nail design should both facilitate insertion and anatomically fit the bone geometry at its final position in order to reduce the risk of stress fractures and malalignments. Due to the nonexistence of suitable commercial software, we developed a software tool for the automated fit assessment of nail designs. Furthermore, we demonstrated that an optimised nail, which fits better at the final position, is also easier to insert. Three-dimensional models of two nail designs and 20 tibiae were used. The fitting was quantified in terms of surface area, maximum distance, sum of surface areas and sum of maximum distances by which the nail was protruding into the cortex. The software was programmed to insert the nail into the bone model and to quantify the fit at defined increment levels. On average, the misfit during the insertion in terms of the four fitting parameters was smaller for the Expert Tibial Nail Proximal bend (476.3 mm2, 1.5 mm, 2029.8 mm2, 6.5 mm) than the Expert Tibial Nail (736.7 mm2, 2.2 mm, 2491.4 mm2, 8.0 mm). The differences were statistically significant (p ≤ 0.05). The software could be used by nail implant manufacturers for the purpose of implant design validation.

Is there a bone-nail specific entry point? automated fit quantification of tibial nail designs during the insertion for six different nail entry points

Intramedullary nailing is the standard fixation method for displaced diaphyseal fractures of tibia. Selection of the correct nail insertion point is important for axial alignment of bone fragments and to avoid iatrogenic fractures. However, the standard entry point (SEP) may not always optimise the bone-nail fit due to geometric variations of bones. This study aimed to investigate the optimal entry for a given bone-nail pair using the fit quantification software tool previously developed by the authors. The misfit was quantified for 20 bones with two nail designs (ETN and ETN-Proximal Bend) related to the SEP and 5 entry points which were 5 mm and 10 mm away from the SEP. The SEP was the optimal entry point for 50% of the bones used. For the remaining bones, the optimal entry point was located 5 mm away from the SEP, which improved the overall fit by 40% on average. However, entry points 10 mm away from the SEP doubled the misfit. The optimised bone-nail fit can be achieved through the SEP and within the range of a 5 mm radius, except posteriorly. The study results suggest that the optimal entry point should be selected by considering the fit during insertion and not only at the final position.

Fully validated LC–MS/MS method for quantification of homocysteine concentrations in samples of human serum : a new approach

Reported homocysteine (HCY) concentrations in human serum show poor concordance amongst laboratories due to endogenous HCY in the matrices used for assay calibrators and QCs. Hence, we have developed a fully validated LC–MS/MS method for measurement of HCY concentrations in human serum samples that addresses this issue by minimising matrix effects. We used small volumes (20 μL) of 2% Bovine Serum Albumin (BSA) as surrogate matrix for making calibrators and QCs with concentrations adjusted for the endogenous HCY concentration in the surrogate matrix using the method of standard additions. To aliquots (20 μL) of human serum samples, calibrators or QCs, were added HCY-d4 (internal standard) and tris-(2-carboxyethyl) phosphine hydrochloride (TCEP) as reducing agent. After protein precipitation, diluted supernatants were injected into the LC–MS/MS. Calibration curves were linear; QCs were accurate (5.6% deviation from nominal), precise (CV% ≤ 9.6%), stable for four freeze–thaw cycles, and when stored at room temperature for 5 h or at −80 °C (27 days). Recoveries from QCs in surrogate matrix or pooled human serum were 91.9 and 95.9%, respectively. There was no matrix effect using 6 different individual serum samples including one that was haemolysed. Our LC–MS/MS method has satisfied all of the validation criteria of the 2012 EMA guideline.

Visual control for automated aircraft collision avoidance systems

This thesis presents a new vision-based decision and control strategy for automated aircraft collision avoidance that can be realistically applied to the See and Avoid problem. The effectiveness of the control strategy positions the research as a major contribution toward realising the simultaneous operation of manned and unmanned aircraft within civilian airspace. Key developments include novel classical and visual predictive control frameworks, and a performance evaluation technique aligned with existing aviation practise and applicable to autonomous systems. The overall approach is demonstrated through experimental results on a small multirotor unmanned aircraft, and through high fidelity probabilistic simulation studies.

Automated counting of the Northern Pacific Sea Star in the Derwent using shape recognition

A novel shape recognition algorithm was developed to autonomously classify the Northern Pacific Sea Star (Asterias amurenis) from benthic images that were collected by the Starbug AUV during 6km of transects in the Derwent estuary. Despite the effects of scattering, attenuation, soft focus and motion blur within the underwater images, an optimal joint classification rate of 77.5% and misclassification rate of 13.5% was achieved. The performance of algorithm was largely attributed to its ability to recognise locally deformed sea star shapes that were created during the segmentation of the distorted images.

Automated sensory data alignment for environmental and epidermal change monitoring

In this paper we present research adapting a state of the art condition-invariant robotic place recognition algorithm to the role of automated inter- and intra-image alignment of sensor observations of environmental and skin change over time. The approach involves inverting the typical criteria placed upon navigation algorithms in robotics; we exploit rather than attempt to fix the limited camera viewpoint invariance of such algorithms, showing that approximate viewpoint repetition is realistic in a wide range of environments and medical applications. We demonstrate the algorithms automatically aligning challenging visual data from a range of real-world applications: ecological monitoring of environmental change, aerial observation of natural disasters including flooding, tsunamis and bushfires and tracking wound recovery and sun damage over time and present a prototype active guidance system for enforcing viewpoint repetition. We hope to provide an interesting case study for how traditional research criteria in robotics can be inverted to provide useful outcomes in applied situations.

Evolutionary game design : automated game design comes of age

The "Humies" awards are an annual competition held in conjunction with the Genetic and Evolutionary Computation Conference (GECCO), in which cash prizes totalling $10,000 are awarded to the most human-competitive results produced by any form of evolutionary computation published in the previous year. This article describes the gold medal-winning entry from the 2012 "Humies" competition, based on the LUDI system for playing, evaluating and creating new board games. LUDI was able to demonstrate human-competitive results in evolving novel board games that have gone on to be commercially published, one of which, Yavalath, has been ranked in the top 2.5% of abstract board games ever invented. Further evidence of human-competitiveness was demonstrated in the evolved games implicitly capturing several principles of good game design, outperforming human designers in at least one case, and going on to inspire a new sub-genre of games.

Hand, foot and mouth disease in China: Evaluating an automated system for the detection of outbreaks

Objective To evaluate the performance of China’s infectious disease automated alert and response system in the detection of outbreaks of hand, foot and mouth (HFM) disease. Methods We estimated size, duration and delay in reporting HFM disease outbreaks from cases notified between 1 May 2008 and 30 April 2010 and between 1 May 2010 and 30 April 2012, before and after automatic alert and response included HFM disease. Sensitivity, specificity and timeliness of detection of aberrations in the incidence of HFM disease outbreaks were estimated by comparing automated detections to observations of public health staff. Findings The alert and response system recorded 106 005 aberrations in the incidence of HFM disease between 1 May 2010 and 30 April 2012 – a mean of 5.6 aberrations per 100 days in each county that reported HFM disease. The response system had a sensitivity of 92.7% and a specificity of 95.0%. The mean delay between the reporting of the first case of an outbreak and detection of that outbreak by the response system was 2.1 days. Between the first and second study periods, the mean size of an HFM disease outbreak decreased from 19.4 to 15.8 cases and the mean interval between the onset and initial reporting of such an outbreak to the public health emergency reporting system decreased from 10.0 to 9.1 days. Conclusion The automated alert and response system shows good sensitivity in the detection of HFM disease outbreaks and appears to be relatively rapid. Continued use of this system should allow more effective prevention and limitation of such outbreaks in China.

MITIE C-Arm - Advanced innovation in education

Aim: In 2013 QUT introduced the Medical Imaging Training Immersive Environment (MITIE) as a virtual reality (VR) platform that allowed students to practice general radiography. The system software has been expanded to now include C-Arm. The aim of this project was to investigate the use of this technology in the pedagogy of undergraduate medical imaging students who have limited to no experience in the use of the C-Arm clinically. Method: The Medical Imaging Training Immersive Environment (MITIE) application provides students with realistic and fully interactive 3D models of C-Arm equipment. As with VR initiatives in other health disciplines (1–2) the software mimics clinical practice as much as possible and uses 3D technology to enhance 3D spatial awareness and realism. The application allows students to set up and expose a virtual patient in a 3D environment as well as creating the resultant “image” for comparison with a gold standard. Automated feedback highlights ways for the student to improve their patient positioning, equipment setup or exposure factors. The students' equipment knowledge was tested using an on line assessment quiz and surveys provided information on the students' pre-clinical confidence scale, with post-clinical data comparisons. Ethical approval for the project was provided by the university ethics panel. Results: This study is currently under way and this paper will present analysis of initial student feedback relating to the perceived value of the application for confidence in a high risk environment (i.e. operating theatre) and related clinical skills development. Further in-depth evaluation is ongoing with full results to be presented. Conclusion: MITIE C-Arm has a development role to play in the pre-clinical skills training for Medical Radiation Science students. It will augment their theoretical understanding prior to their clinical experience. References 1. Bridge P, Appleyard R, Ward J, Phillips R, Beavis A. The development and evaluation of a virtual radiotherapy treatment machine using an immersive visualisation environment. Computers and Education 2007; 49(2): 481–494. 2. Gunn T, Berry C, Bridge P et al. 3D Virtual Radiography: Development and Initial Feedback. Paper presented at the 10th Annual Scientific Meeting of Medical Imaging and Radiation Therapy, March 2013 Hobart, Tasmania.

A nationwide web-based automated system for early outbreak detection and rapid response in China

Timely reporting, effective analyses and rapid distribution of surveillance data can assist in detecting the aberration of disease occurrence and further facilitate a timely response. In China, a new nationwide web-based automated system for outbreak detection and rapid response was developed in 2008. The China Infectious Disease Automated-alert and Response System (CIDARS) was developed by the Chinese Center for Disease Control and Prevention based on the surveillance data from the existing electronic National Notifiable Infectious Diseases Reporting Information System (NIDRIS) started in 2004. NIDRIS greatly improved the timeliness and completeness of data reporting with real time reporting information via the Internet. CIDARS further facilitates the data analysis, aberration detection, signal dissemination, signal response and information communication needed by public health departments across the country. In CIDARS, three aberration detection methods are used to detect the unusual occurrence of 28 notifiable infectious diseases at the county level and to transmit that information either in real-time or on a daily basis. The Internet, computers and mobile phones are used to accomplish rapid signal generation and dissemination, timely reporting and reviewing of the signal response results. CIDARS has been used nationwide since 2008; all Centers for Disease Control and Prevention (CDC) in China at the county, prefecture, provincial and national levels are involved in the system. It assists with early outbreak detection at the local level and prompts reporting of unusual disease occurrences or potential outbreaks to CDCs throughout the country.