Predictability in high-stakes examinations: students' perspectives on a perennial assessment dilemma

Key debates within educational assessment continuously encourage us to reflect on the design, delivery and implementation of examination systems as well as their relevance to students. In more recent times, such reflections have also required a rethinking of who is authoritative about assessment issues and whose views we seek in order to better understand these perennial assessment dilemmas. This paper considers one such dilemma, predictability in high-stakes assessment, and presents students’ perspectives on this issue. The context is the Irish Leaving Certificate (LC) taken by upper secondary students (aged between 16 and 18) in order (mainly) to enter tertiary-level education. The data come from 13 group interviews with 81 students across a range of schools in Ireland. Listening to students about complex, high-stakes examining problems has a limited history within the educational assessment literature. The findings from the study address this shortcoming and depict how students’ insightful reflections can improve our understanding of these dilemmas. Further, students are more than able to reflect on their own situations with regard to high stakes examining contexts and have important contributions to make to our fuller understanding of those elements that will promote high quality and fair assessment.

A qualitative study of curriculum and instructional efforts in relation to high stakes federal and district policies in one low-performing elementary school

Les enjeux liés aux politiques éducatives ont considérablement changé au cours des dernières décennies. Ces changements sont liés, entre autres, à l’accroissement de l’imputabilité et de la reddition de compte qui est devenue une caractéristique importante des réformes curriculaires et pédagogiques. Les politiques à enjeux élevés exercent une pression énorme sur les districts et les écoles états-unienne afin qu’ils augmentent le rendement des élèves en utilisant des systèmes de conséquences (Hall & Ryan, 2011; Loeb & Strunk, 2007). Ces politiques envoient de puissants messages sur l'importance de certaines matières scolaires au détriment d'autres - circonscrivant les exigences en termes de compétences et de connaissances. La langue maternelle d’enseignement et les mathématiques sont devenues des mesures centrales sur lesquelles reposent l’évaluation et le degré de performance des districts et des écoles. Conséquemment, les administrateurs de districts et les directions d’écoles ont souvent recours à des réformes curriculaires et pédagogiques comme moyen d'augmenter le rendement des élèves dans les matières scolaires visées par ces politiques. Les politiques contraignent les acteurs scolaires de concentrer les ressources sur les programmes curriculaires et les évaluations, le développement professionnel, et la prise de décision pilotée par les données (Anagnostopoulos & Ruthledge, 2007; Honig & Hatch, 2004; Spillane, Diamond, et al., 2002; Weitz White & Rosenbaum, 2008). Cette thèse examine la manière dont les politiques à enjeux élevés opèrent quotidiennement dans les interactions et les pratiques au sein des écoles. Nous analysons plus particulièrement les différents messages provenant de la politique transmis aux acteurs scolaires sur les manières d'apporter des changements substantiels dans le curriculum et l'enseignement. Nous élargissons l’analyse en prenant en compte le rôle des administrateurs de district ainsi que des partenaires universitaires qui façonnent également la manière dont certains aspects des messages provenant des politiques sont transmis, négociés et/ou débattus et d’autres sont ignorés (Coburn & Woulfin, 2012). En utilisant l’analyse de discours, nous examinons le rôle du langage comme constituant et médiateur des interactions sociales entre les acteurs scolaires et d’autres parties prenantes. De telles analyses impliquent une investigation approfondie d’un nombre d’étude de cas limité. Les données utilisées dans cette thèse ont été colligées dans une école primaire états-unienne du mid-West. Cette étude de cas fait partie d’une étude longitudinale de quatre ans qui comprenait huit écoles dans les milieux urbains entre 1999 et 2003 (Distributed Leadership Studies, http://www.distributedleadership.org). La base de données analysée inclut des observations de réunions formelles et des entrevues auprès des administrateurs du district, des partenaires universitaires, de la direction d’école et des enseignants. En plus de l’introduction et de la problématique (chapitre 1) et de discussion et conclusion (chapitre 5), cette thèse comprend un ensemble de trois articles interdépendants. Dans le premier article (chapitre 2), nous effectuons une recension des écrits portant sur le domaine de l’implantation de politiques (policy implementation) et la complexité des relations locales, nationales et internationales dans les systèmes éducatifs. Pour démystifier cette complexité, nous portons une attention particulière à la construction de sens des acteurs scolaires comme étant une dimension clé du processus de mise en œuvre des réformes. Dans le deuxième article (chapitre 3), nous cherchons à comprendre les processus sociaux qui façonnent les réponses stratégiques des acteurs scolaires à l’égard des politiques du district et de l’état et en lien avec la mise en œuvre d’un curriculum prescrit en mathématiques. Plus particulièrement, nous explorons les différentes situations dans lesquelles les acteurs scolaires argumentent au sujet des changements curriculaires et pédagogiques proposés par les administrateurs de district et des partenaires universitaires afin d’augmenter les résultats scolaires en mathématiques dans une école à faible performance. Dans le troisième article (chapitre 4), nous cherchons à démystifier les complexités liées à l’amélioration de l’enseignement dans un environnement de politiques à enjeux élevés. Pour ce faire, nous utilisons l'interaction entre les notions d'agentivité et la structure afin d'analyser la manière dont les conceptions d’imputabilité et les idées qui découlent de l'environnement politique et les activités quotidiennes jouent dans les interactions entre les acteurs scolaires concernant sur l’enseignement de la langue maternelle. Nous explorons trois objectifs spécifiques : 1) la manière dont les politiques à enjeux élevés façonnent les éléments de l’enseignement qui sont reproduits et ceux qui sont transformés au fil du temps ; 2) la manière dont la compréhension des leaders de l’imputabilité façonne les aspects des messages politiques que les acteurs scolaires remarquent à travers les interactions et les conversations et 3) la manière les acteurs scolaires portent une attention particulière à certaines messages au détriment d’autres. Dans le dernier chapitre de cette thèse, nous discutons les forces et les limites de l’analyse secondaire de données qualitatives, les implications des résultats pour le domaine d’études de l’implantation de politiques et les pistes futures de recherches.

Rarely selected distractors in high stakes medical multiple-choice examinations and their recognition by item authors: A simulation and survey

Background Many medical exams use 5 options for multiple choice questions (MCQs), although the literature suggests that 3 options are optimal. Previous studies on this topic have often been based on non-medical examinations, so we sought to analyse rarely selected, 'non-functional' distractors (NF-D) in high stakes medical examinations, and their detection by item authors as well as psychometric changes resulting from a reduction in the number of options. Methods Based on Swiss Federal MCQ examinations from 2005-2007, the frequency of NF-D (selected by <1% or <5% of the candidates) was calculated. Distractors that were chosen the least or second least were identified and candidates who chose them were allocated to the remaining options using two extreme assumptions about their hypothetical behaviour: In case rarely selected distractors were eliminated, candidates could randomly choose another option - or purposively choose the correct answer, from which they had originally been distracted. In a second step, 37 experts were asked to mark the least plausible options. The consequences of a reduction from 4 to 3 or 2 distractors - based on item statistics or on the experts' ratings - with respect to difficulty, discrimination and reliability were modelled. Results About 70% of the 5-option-items had at least 1 NF-D selected by <1% of the candidates (97% for NF-Ds selected by <5%). Only a reduction to 2 distractors and assuming that candidates would switch to the correct answer in the absence of a 'non-functional' distractor led to relevant differences in reliability and difficulty (and to a lesser degree discrimination). The experts' ratings resulted in slightly greater changes compared to the statistical approach. Conclusions Based on item statistics and/or an expert panel's recommendation, the choice of a varying number of 3-4 (or partly 2) plausible distractors could be performed without marked deteriorations in psychometric characteristics.

Approaches to Ensure Quality Standards for Standardized/ Simulated Patient Performance in High Stakes Objective Structured Clinical Exams (OSCEs)

Introduction To meet the quality standards for high-stakes OSCEs, it is necessary to ensure high quality standardized performance of the SPs involved.[1] One of the ways this can be assured is through the assessment of the quality of SPs` performance in training and during the assessment. There is some literature concerning validated instruments that have been used to assess SP performance in formative contexts but very little related to high stakes contexts.[2], [3], [4]. Content and structure During this workshop different approaches to quality control for SPs` performance, developed in medicine, pharmacy and nursing OSCEs, will be introduced. Participants will have the opportunity to use these approaches in simulated interactions. Advantages and disadvantages of these approaches will be discussed. Anticipated outcomes By the end of this session, participants will be able to discuss the rationale for quality control of SPs` performance in high stakes OSCEs, outline key factors in creating strategies for quality control, identify various strategies for assuring quality control, and reflect on applications to their own practice. Who should attend The workshop is designed for those interested in quality assurance of SP performance in high stakes OSCEs. Level All levels are welcome. References Adamo G. 2003. Simulated and standardized patients in OSCEs: achievements and challenges:1992-2003. Med Teach. 25(3), 262- 270. Wind LA, Van Dalen J, Muijtjens AM, Rethans JJ. Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Med Educ 2004, 38(1):39-44. Bouter S, van Weel-Baumgarten E, Bolhuis S. Construction and validation of the Nijmegen Evaluation of the Simulated Patient (NESP): Assessing Simulated Patients' ability to role-play and provide feedback to students. Acad Med: Journal of the Association of American Medical Colleges 2012. May W, Fisher D, Souder D: Development of an instrument to measure the quality of standardized/simulated patient verbal feedback. Med Educ 2012, 2(1).

Quality Control for SP Performance Regarding Patient Portrayal in a High-Stakes OSCE

Introduction Since the quality of patient portrayal of standardized patients (SPs) during an Objective Structured Clinical Exam (OSCE) has a major impact on the reliability and validity of the exam, quality control should be initiated. Literature about quality control of SP’s performance focuses on feedback [1, 2] or completion of checklists [3, 4]. Since we did not find a published instrument meeting our needs for the assessment of patient portrayal, we developed such an instrument after being inspired by others [5] and used it in our high-stakes exam. Methods SP trainers from all five Swiss medical faculties collected and prioritized quality criteria for patient portrayal. Items were revised with the partners twice, based on experiences during OSCEs. The final instrument contains 14 criteria for acting (i.e. adequate verbal and non-verbal expression) and standardization (i.e. verbatim delivery of the first sentence). All partners used the instrument during a high-stakes OSCE. Both, SPs and trainers were introduced to the instrument. The tool was used in training (more than 100 observations) and during the exam (more than 250 observations). FAIR_OSCE The list of items to assess the quality of the simulation by SPs was primarily developed and used to provide formative feedback to the SPs in order to help them to improve their performance. It was therefore named “Feedbackstruckture for the Assessment of Interactive Role play in Objective Structured Clinical Exams (FAIR_OSCE). It was also used to assess the quality of patient portrayal during the exam. The results were calculated for each of the five faculties individually. Formative evaluation was given to the five faculties with individual feedback without revealing results of other faculties other than overall results. Results High quality of patient portrayal during the exam was documented. More than 90% of SP performances were rated to be completely correct or sufficient. An increase in quality of performance between training and exam was noted. In example the rate of completely correct reaction in medical tests increased from 88% to 95%. 95% completely correct reactions together with 4% sufficient reactions add up to 99% of the reactions meeting the requirements of the exam. SP educators using the instrument reported an augmentation of SPs performance induced by the use of the instrument. Disadvantages mentioned were high concentration needed to explicitly observe all criteria and cumbersome handling of the paper-based forms. Conclusion We were able to document a very high quality of SP performance in our exam. The data also indicate that our training is effective. We believe that the high concentration needed using the instrument is well invested, considering the observed augmentation of performance. The development of an iPad based application for the form is planned to address the cumbersome handling of the paper.

Illinois statewide high school testing program. unit on evaluation.

Mode of access: Internet.

Report of the Tennessee State high school testing program,

References: p. 92.

Base heating investigations on a 1/10 scale model Centaur stage using short-duration high altitude testing techniques. Contract no. NAS 3-3227.

Mode of access: Internet.

The high stakes game of international standard setting.

"October 1987"--P. [4].

Medical students' personal experience of high-stakes failure:case studies using interpretative phenomenological analysis

Abstract (provisional): Background Failing a high-stakes assessment at medical school is a major event for those who go through the experience. Students who fail at medical school may be more likely to struggle in professional practice, therefore helping individuals overcome problems and respond appropriately is important. There is little understanding about what factors influence how individuals experience failure or make sense of the failing experience in remediation. The aim of this study was to investigate the complexity surrounding the failure experience from the student’s perspective using interpretative phenomenological analysis (IPA). Methods The accounts of 3 medical students who had failed final re-sit exams, were subjected to in-depth analysis using IPA methodology. IPA was used to analyse each transcript case-by-case allowing the researcher to make sense of the participant’s subjective world. The analysis process allowed the complexity surrounding the failure to be highlighted, alongside a narrative describing how students made sense of the experience. Results The circumstances surrounding students as they approached assessment and experienced failure at finals were a complex interaction between academic problems, personal problems (specifically finance and relationships), strained relationships with friends, family or faculty, and various mental health problems. Each student experienced multi-dimensional issues, each with their own individual combination of problems, but experienced remediation as a one-dimensional intervention with focus only on improving performance in written exams. What these students needed to be included was help with clinical skills, plus social and emotional support. Fear of termination of the their course was a barrier to open communication with staff. Conclusions These students’ experience of failure was complex. The experience of remediation is influenced by the way in which students make sense of failing. Generic remediation programmes may fail to meet the needs of students for whom personal, social and mental health issues are a part of the picture.

High Stakes and No Takers: The Impact of Florida Comprehensive Assessment Test (FCAT) Writing on Students’ and Teachers’ Perceptions of Writing

This ethnographic case study of ten students and their teachers concludes that the state writing test had a negative impact on students’ and teachers’ perceptions of writing in four categories: strengths and weaknesses in student writing, self-assessment of writing skills, factors impacting test scores, and motivation and attitudes toward writing.

UNDERSTANDING ESSAY RATING AS A SOCIALLY MEDIATED ACTIVITY: THE CASE OF A HIGH-STAKES ENGLISH TEST IN CHINA

Most essay rating research in language assessment has examined human raters’ essay rating as a cognitive process, thus overlooking or oversimplifying the interaction between raters and sociocultural contexts. Given that raters are social beings, their practices have social meanings and consequences. Hence it is important to situate essay rating within its sociocultural context for a more meaningful understanding. Drawing on Engeström’s (1987, 2001) cultural-historical activity theory (CHAT) framework with a sociocultural perspective, this study reconceptualized essay rating as a socially mediated activity with both cognitive (individual raters’ goal-directed decision-making actions) and social layers (raters’ collective object-oriented essay rating activity at related settings). In particular, this study explored raters’ essay rating at one provincial rating centre in China within the context of a high-stakes university entrance examination, the National Matriculation English Test (NMET). This study adopted a multiple-method multiple-perspective qualitative case study design. Think-aloud protocols, stimulated recalls, interviews, and documents served as the data sources. This investigation involved 25 participants at two settings (rating centre and high schools), including rating centre directors, team leaders, NMET essay raters who were high school teachers, and school principals and teaching colleagues of these essay raters. Data were analyzed using Strauss and Corbin’s (1990) open and axial coding techniques, and CHAT for data integration. The findings revealed the interaction between raters and the NMET sociocultural context. Such interaction can be understood through a surface structure (cognitive layer) and a deep structure (social layer) concerning how raters assessed NMET essays, where the surface structure reflected the “what” and the deep structure explained the “how” and “why” in raters’ decision-making. This study highlighted the roles of goals and rules in rater decision-making, rating tensions and raters’ solutions, and the relationship between essay rating and teaching. This study highlights the value of a sociocultural view to essay rating research, demonstrates CHAT as a sociocultural approach to investigate essay rating, and proposes a direction for future washback research on the effect of essay rating. This study also provides support for NMET rating practices that can potentially bring positive washback to English teaching in Chinese high schools.

UNDERSTANDING ESSAY RATING AS A SOCIALLY MEDIATED ACTIVITY: THE CASE OF A HIGH-STAKES ENGLISH TEST IN CHINA

Thesis (Ph.D, Education) -- Queen's University, 2016-09-22 22:05:24.246

Creative pedagogies and the contemporary school classroom

This chapter explores the ‘creative pedagogies’ of imaginative teaching and learning and the development of creative capacities in formal schooling. It considers how educational policies enable and constrain creative thinking as students learn through play and experimentation. The chapter outlines two classroom based case studies involving creative media technologies in Queensland, Australia. It argues that Queensland’s ‘open’ system of curriculum development enabled the two schools to implement transdisciplinary ‘rich tasks’ for the students. However, there are constraints related to the social mediation of creativity and the influence of high stakes testing in foregrounding ‘value’ and ‘purpose’ in learning.

A controlled rapid approach to curriculum change : addressing the pressures of increased test-based accountability on schools

Increasingly, large amounts of public and private money are being invested in education and as a result, schools are becoming more accountable to stakeholders for this financial input. In terms of the curriculum, governments worldwide are frequently tying school funding to students‟ and schools‟ academic performances, which are monitored through high-stakes testing programs. To accommodate the resultant pressures from these testing initiatives, many principals are re-focussing their school‟s curriculum on the testing requirements. Such a re-focussing, which was examined critically in this thesis, constituted an externally facilitated rapid approach to curriculum change. In line with previously enacted change theories and recommendations from these, curriculum change in schools has tended to be a fairly slow, considered, collaborative process that is facilitated internally by a deputy-principal (curriculum). However, theoretically based research has shown that such a process has often proved to be difficult and very rarely successful. The present study reports and theorises the experiences of an externally facilitated process that emerged from a practitioner model of change. This case study of the development of the controlled rapid approach to curriculum change began by establishing the reasons three principals initiated curriculum change and why they then engaged an outsider to facilitate the process. It also examined this particular change process from the perspectives of the research participants. The investigation led to the revision of the practitioner model as used in the three schools and challenged the current thinking about the process of school curriculum change. The thesis aims to offer principals and the wider education community an alternative model for consideration when undertaking curriculum change. Finally, the thesis warns that, in the longer term, the application of study‟s revised model (the Controlled Rapid Approach to Curriculum Change [CRACC] Model) may have less then desirable educational consequences.