Assessing the evidence: Student response system versus computer based testing for undertaking multiple choice question assessment in undergraduate nursing education

There is a dearth of evidence focusing on student preferences for computer-based testing versus
testing via student response systems for summative assessment in undergraduate education.
This quantitative study compared the preference and acceptability of computer-based testing
and a student response system for completing multiple choice questions in undergraduate
nursing education. After using both computer-based testing and a student response system to
complete multiple choice questions, 192 first year undergraduate nursing students rated their
preferences and attitudes towards using computer-based testing and a student response system.
Results indicated that seventy four percent felt the student response system was easy to use.
Fifty six percent felt the student response system took more time than the computer-based testing
to become familiar with. Sixty Percent felt computer-based testing was more users friendly.
Seventy Percent of students would prefer to take a multiple choice question summative exam
via computer-based testing, although Fifty percent would be happy to take using student response
system. Results are useful for undergraduate educators in relation to student’s preference
for using computer-based testing or student response system to undertake a summative
multiple choice question exam

Improving E-Assessment Systems With A focus On Adaptive Feedback And Marking

Institutions involved in the provision of tertiary education across Europe are feeling the pinch. European universities, and other higher education (HE) institutions, must operate in a climate where the pressure of government spending cuts (Garben, 2012) is in stark juxtaposition to the EU’s strategy to drive forward and maintain a growth of student numbers in the sector (eurostat, 2015).

In order to remain competitive, universities and HE institutions are making ever-greater use of electronic assessment (E-Assessment) systems (Chatzigavriil et all, 2015; Ferrell, 2012). These systems are attractive primarily because they offer a cost-effect and scalable approach for assessment. In addition to scalability, they also offer reliability, consistency and impartiality; furthermore, from the perspective of a student they are most popular because they can offer instant feedback (Walet, 2012).

There are disadvantages, though.

First, feedback is often returned to a student immediately on competition of their assessment. While it is possible to disable the instant feedback option (this is often the case during an end of semester exam period when assessment scores must be can be ratified before release), however, this option tends to be a global ‘all on’ or ‘all off’ configuration option which is controlled centrally rather than configurable on a per-assessment basis.

If a formative in-term assessment is to be taken by multiple groups of
students, each at different times, this restriction means that answers to each question will be disclosed to the first group of students undertaking the assessment. As soon as the answers are released “into the wild” the academic integrity of the assessment is lost for subsequent student groups.

Second, the style of feedback provided to a student for each question is often limited to a simple ‘correct’ or ‘incorrect’ indicator. While this type of feedback has its place, it often does not provide a student with enough insight to improve their understanding of a topic that they did not answer correctly.

Most E-Assessment systems boast a wide range of question types including Multiple Choice, Multiple Response, Free Text Entry/Text Matching and Numerical questions. The design of these types of questions is often quite restrictive and formulaic, which has a knock-on effect on the quality of feedback that can be provided in each case.

Multiple Choice Questions (MCQs) are most prevalent as they are the most prescriptive and therefore most the straightforward to mark consistently. They are also the most amenable question types, which allow easy provision of meaningful, relevant feedback to each possible outcome chosen.
Text matching questions tend to be more problematic due to their free text entry nature. Common misspellings or case-sensitivity errors can often be accounted for by the software but they are by no means fool proof, as it is very difficult to predict in advance the range of possible variations on an answer that would be considered worthy of marks by a manual marker of a paper based equivalent of the same question.

Numerical questions are similarly restricted. An answer can be checked for accuracy or whether it is within a certain range of the correct answer, but unless it is a special purpose-built mathematical E-Assessment system the system is unlikely to have computational capability and so cannot, for example, account for “method marks” which are commonly awarded in paper-based marking.

From a pedagogical perspective, the importance of providing useful formative feedback to students at a point in their learning when they can benefit from the feedback and put it to use must not be understated (Grieve et all, 2015; Ferrell, 2012).

In this work, we propose a number of software-based solutions, which will overcome the limitations and inflexibilities of existing E-Assessment systems.

An evaluation of the Socrative (personal response system) app for increasing student engagement and learning in an undergraduate psychology curriculum.

Personal response systems using hardware such as 'clickers' have been around for some time, however their use is often restricted to multiple choice questions (MCQs) and they are therefore used as a summative assessment tool for the individual student. More recent innovations such as 'Socrative' have removed the need for specialist hardware, instead utilising web-based technology and devices common to students, such as smartphones, tablets and laptops. While improving the potential for use in larger classrooms, this also creates the opportunity to pose more engaging open-response questions to students who can 'text in' their thoughts on questions posed in class. This poster will present two applications of the Socrative system in an undergraduate psychology curriculum which aimed to encourage interactive engagement with course content using real-time student responses and lecturer feedback. Data is currently being collected and result will be presented at the conference.
The first application used Socrative to pose MCQs at the end of two modules (a level one Statistics module and level two Individual Differences Psychology module, class size N≈100), with the intention of helping students assess their knowledge of the course. They were asked to rate their self-perceived knowledge of the course on a five-point Likert scale before and after completing the MCQs, as well as their views on the value of the revision session and any issues that had with using the app. The online MCQs remained open between the lecture and the exam, allowing students to revisit the questions at any time during their revision.
This poster will present data regarding the usefulness of the revision MCQs, the metacognitive effect of the MCQs on student's judgements of learning (pre vs post MCQ testing), as well as student engagement with the MCQs between the revision session and the examination. Student opinions on the use of the Socrative system in class will also be discussed.
The second application used Socrative to facilitate a flipped classroom lecture on a level two 'Conceptual Issues in Psychology' module, class size N≈100). The content of this module requires students to think critically about historical and contemporary conceptual issues in psychology and the philosophy of science. Students traditionally struggle with this module due to the emphasis on critical thinking skills, rather than simply the retention of concrete knowledge. To prepare students for the written examination, a flipped classroom lecture was held at the end of the semester. Students were asked to revise their knowledge of a particular area of Psychology by assigned reading, and were told that the flipped lecture would involve them thinking critically about the conceptual issues found in this area. They were informed that questions would be posed by the lecturer in class, and that they would be asked to post their thoughts using the Socrative app for a class discussion. The level of preparation students engaged in for the flipped lecture was measured, as well as qualitative opinions on the usefulness of the session. This poster will discuss the level of student engagement with the flipped lecture, both in terms of preparation for the lecture, and engagement with questions posed during the lecture, as well as the lecturer's experience in facilitating the flipped classroom using the Socrative platform.

Investigating high-achieving sixth grade students' erroneous answers on the angle concept in Yozgat

The angle concept is a multifaceted concept having static and dynamic definitions. The static definition of the angle refers to “the space between two rays” or “the intersection of two rays at the same end point” (Mitchelmore & White, 1998), whereas the dynamic definition of the angle concept highlights that the size of angle is the amount of rotation in direction (Fyhn, 2006). Since both definitions represent two diverse situations and have unique limitations (Henderson & Taimina, 2005), students may hold misconceptions about the angle concept. In this regard, the aim of this research was to explore high achievers’ knowledge regarding the definition of the angle concept as well as to investigate their erroneous answers on the angle concept.

104 grade 6 students drawn from four well-established elementary schools of Yozgat, Turkey were participated in this research. All participants were selected via a purposive sampling method and their mathematics grades were 4 or 5 out of 5, and. Data were collected through four questions prepared by considering the learning competencies set out in the grade 6 curriculum in Turkey and the findings of previous studies whose purposes were to identify students’ misconceptions of the angle concept. The findings were analyzed by two researchers, and their inter-rater agreement was calculated as 0.91, or almost perfect. Thereafter, coding discrepancies were resolved, and consensus was established.

The angle concept is a multifaceted concept having static and dynamic definitions.The static definition of the angle refers to “the space between two rays” or“the intersection of two rays at the same end point” (Mitchelmore & White, 1998), whereas the dynamicdefinition of the angle concept highlights that the size of angle is the amountof rotation in direction (Fyhn, 2006). Since both definitionsrepresent two diverse situations and have unique limitations (Henderson & Taimina, 2005), students may holdmisconceptions about the angle concept. In this regard, the aim of thisresearch was to explore high achievers’ knowledge regarding the definition ofthe angle concept as well as to investigate their erroneous answers on theangle concept.

Is Our Students Learning - QuestionMark: Extending E-Assessment Systems for Adapative Marking and Feedback

Our key contribution is a flexible, automated marking system that adds desirable functionality to existing E-Assessment systems. In our approach, any given E-Assessment system is relegated to a data-collection mechanism, whereas marking and the generation and distribution of personalised per-student feedback is handled separately by our own system. This allows content-rich Microsoft Word feedback documents to be generated and distributed to every student simultaneously according to a per-assessment schedule.

The feedback is adaptive in that it corresponds to the answers given by the student and provides guidance on where they may have gone wrong. It is not limited to simple multiple choice which are the most prescriptive question type offered by most E-Assessment Systems and as such most straightforward to mark consistently and provide individual per-alternative feedback strings. It is also better equipped to handle the use of mathematical symbols and images within the feedback documents which is more flexible than existing E-Assessment systems, which can only handle simple text strings.

As well as MCQs the system reliably and robustly handles Multiple Response, Text Matching and Numeric style questions in a more flexible manner than Questionmark: Perception and other E-Assessment Systems. It can also reliably handle multi-part questions where the response to an earlier question influences the answer to a later one and can adjust both scoring and feedback appropriately.

New question formats can be added at any time provided a corresponding marking method conforming to certain templates can also be programmed. Indeed, any question type for which a programmatic method of marking can be devised may be supported by our system. Furthermore, since the student’s response to each is question is marked programmatically, our system can be set to allow for minor deviations from the correct answer, and if appropriate award partial marks.

Simulation using novel equipment designed to explain spirometric abnormalities in respiratory disease enhances learning in higher cognitive domains

Simulation of disorders of respiratory mechanics shown by spirometry provides insight into the pathophysiology of disease but some clinically important disorders have not been simulated and none have been formally evaluated for education. We have designed simple mechanical devices which, along with existing simulators, enable all the main dysfunctions which have diagnostic value in spirometry to be simulated and clearly explained with visual and haptic feedback. We modelled the airways as Starling resistors by a clearly visible mechanical action to simulate intra- and extra-thoracic obstruction. A narrow tube was used to simulate fixed large airway obstruction and inelastic bands to simulate restriction. We hypothesized that using simulators whose action explains disease promotes learning especially in higher domain educational objectives. The main features of obstruction and restriction were correctly simulated. Simulation of variable extra-thoracic obstruction caused blunting and plateauing of inspiratory flow, and simulation of intra-thoracic obstruction caused limitation of expiratory flow with marked dynamic compression. Multiple choice tests were created with questions allocated to lower (remember and understand) or higher cognitive domains (apply, analyse and evaluate). In a cross-over design, overall mean scores increased after 1½ h simulation spirometry (43-68 %, effect size 1.06, P < 0.0001). In higher cognitive domains the mean score was lower before and increased further than lower domains (Δ 30 vs 20 %, higher vs lower effect size 0.22, P < 0.05). In conclusion, the devices successfully simulate various patterns of obstruction and restriction. Using these devices medical students achieved marked enhancement of learning especially in higher cognitive domains.

The Value of Reducing Cancer Risks at Contaminated Sites: Are More Knowledgeable People Willing to Pay More?

We use conjoint choice questions to investigate people's tastes for cancer risk reductions and income in the context of public programs that would provide for remediation at abandoned industrial contaminated sites. Our survey was self-administered using the computer by persons living in the vicinity of an important contaminated site on the Italian National Priority List. The value of a prevented case of cancer is €2.6 million, but this figure does vary with income, perceived exposure to contaminants, and respondent opinions about priorities that should be pursued by cleanup programs. © 2011 Society for Risk Analysis.

Precision Teaching in IV Fluid Prescribing and Patient Safety

OBJECTIVES: Precision Teaching (PT) has been shown to be an effective intervention to assess teaching method effectiveness and evaluate learning outcomes. SAFMEDS (Say All Fast Minute Every Day Shuffled) are a practice/assessment procedure within the PT framework to assist learning and fluency. We explored the effects of a brief intervention with PT, to impart high frequency performance in safe intravenous fluid prescription in a group of final year undergraduate medical students.
METHODS: 133 final year undergraduate medical students completed a multiple choice question (MCQ) test on safe IV fluid prescription at the beginning and end of the study. The control group (n= 76) of students were taught using a current standardized teaching method. Students allocated to the intervention arm of the study were additionally instructed on PT and the use of SAFMEDS. The study group (n = 57) received 50 SAFMEDS cards containing information on the principles of IV fluid prescription scenarios. These students were trained/tested twice per day for 1 minute.
RESULTS: Interim analysis showed that the study group displayed an improvement in fluency and accuracy as the study progressed. There was a statistically significant improvement in MCQ performance for the PT group compared with the control group between the beginning and end of the study (35% vs 15%).
CONCLUSION: These results suggest PT employing SAFMEDS is an effective method for improving fluency, accuracy and patient safety in intravenous fluid prescribing amongst undergraduate medical students.

Applied behaviour analysis: What do teachers of students with autism spectrum disorder know

With rising numbers of school-aged children with autism educated in mainstream classrooms and applied behaviour analysis (ABA) considered the basis of best practice, teachers’ knowledge in this field has become a key concern for inclusion. Self-reported knowledge of ABA of special needs teachers (n=165) was measured and compared to their actual knowledge of ABA demonstrated in accurate responses to a multiple-choice test. Findings reported here show that teachers’ self-perceived knowledge exceeded actual knowledge and that actual knowledge of ABA was not related to training received by government agency. Implications for teacher training are discussed.

Perception of Filtered Speech by Children with Developmental Dyslexia and Children with Specific Language Impairments

Here we use two filtered speech tasks to investigate children’s processing of slow (<4 Hz) versus faster (∼33 Hz) temporal modulations in speech. We compare groups of children with either developmental dyslexia (Experiment 1) or speech and language impairments (SLIs, Experiment 2) to groups of typically-developing (TD) children age-matched to each disorder group. Ten nursery rhymes were filtered so that their modulation frequencies were either low-pass filtered (<4 Hz) or band-pass filtered (22 – 40 Hz). Recognition of the filtered nursery rhymes was tested in a picture recognition multiple choice paradigm. Children with dyslexia aged 10 years showed equivalent recognition overall to TD controls for both the low-pass and band-pass filtered stimuli, but showed significantly impaired acoustic learning during the experiment from low-pass filtered targets. Children with oral SLIs aged 9 years showed significantly poorer recognition of band pass filtered targets compared to their TD controls, and showed comparable acoustic learning effects to TD children during the experiment. The SLI samples were also divided into children with and without phonological difficulties. The children with both SLI and phonological difficulties were impaired in recognizing both kinds of filtered speech. These data are suggestive of impaired temporal sampling of the speech signal at different modulation rates by children with different kinds of developmental language disorder. Both SLI and dyslexic samples showed impaired discrimination of amplitude rise times. Implications of these findings for a temporal sampling framework for understanding developmental language disorders are discussed.

Applied behavior analysis: What do teachers of students with autism spectrum disorder know?

With rising numbers of school-aged children with autism educated in mainstream classroomsand applied behavior analysis (ABA) considered the basis of best practice, teachers’ knowledgein this field has become a key concern for inclusion. Self-reported knowledge of ABA of specialneeds teachers (n=165) was measured and compared to their actual knowledge of ABAdemonstrated in accurate responses to a multiple-choice test. Findings reported here show thatteachers’ self-perceived knowledge exceeded actual knowledge and that actual knowledge ofABA was not

A literature review to Examine the Validity and Reliability of group poster presentations as an assessment method of a module in an undergraduate nursing degree programme

The introduction of a poster presentation as a formative assessment method over a multiple choice examination after the first phase of a three phase “health and well-being” module in an undergraduate nursing degree programme was greeted with a storm of criticism from fellow lecturers stating that poster presentations are not valid or reliable and totally irrelevant to the assessment of learning in the module. This paper seeks to investigate these criticisms by investigating the literature regarding producing nurses fit for practice, nurse curriculum development and wider nurse education, the purpose of assessment, validity and reliability to critically evaluate the poster presentation as a legitimate assessment method for these aims.