Building academic resilience: A longitudinal study of Master of Teaching students’ perceptions of written feedback on high stakes assessment

This paper reports on a two-year study of tertiary education students’ perceptions of written feedback on assessment. Contextualised in a regional Australian university and drawing on a cohort of Master of Teaching pre-service teachers, a survey approach was used to collect data from participants in both years of their graduate-level entry program. The paper contributes to the emerging body of literature on students’ perceptions of written feedback on assessment by discussing four themes that emerged from the study. It concludes with a number of recommendations for teacher educators to enhance current feedback practices.

Predictability in high-stakes examinations: students' perspectives on a perennial assessment dilemma

Key debates within educational assessment continuously encourage us to reflect on the design, delivery and implementation of examination systems as well as their relevance to students. In more recent times, such reflections have also required a rethinking of who is authoritative about assessment issues and whose views we seek in order to better understand these perennial assessment dilemmas. This paper considers one such dilemma, predictability in high-stakes assessment, and presents students’ perspectives on this issue. The context is the Irish Leaving Certificate (LC) taken by upper secondary students (aged between 16 and 18) in order (mainly) to enter tertiary-level education. The data come from 13 group interviews with 81 students across a range of schools in Ireland. Listening to students about complex, high-stakes examining problems has a limited history within the educational assessment literature. The findings from the study address this shortcoming and depict how students’ insightful reflections can improve our understanding of these dilemmas. Further, students are more than able to reflect on their own situations with regard to high stakes examining contexts and have important contributions to make to our fuller understanding of those elements that will promote high quality and fair assessment.

Medical students' personal experience of high-stakes failure:case studies using interpretative phenomenological analysis

Abstract (provisional): Background Failing a high-stakes assessment at medical school is a major event for those who go through the experience. Students who fail at medical school may be more likely to struggle in professional practice, therefore helping individuals overcome problems and respond appropriately is important. There is little understanding about what factors influence how individuals experience failure or make sense of the failing experience in remediation. The aim of this study was to investigate the complexity surrounding the failure experience from the student’s perspective using interpretative phenomenological analysis (IPA). Methods The accounts of 3 medical students who had failed final re-sit exams, were subjected to in-depth analysis using IPA methodology. IPA was used to analyse each transcript case-by-case allowing the researcher to make sense of the participant’s subjective world. The analysis process allowed the complexity surrounding the failure to be highlighted, alongside a narrative describing how students made sense of the experience. Results The circumstances surrounding students as they approached assessment and experienced failure at finals were a complex interaction between academic problems, personal problems (specifically finance and relationships), strained relationships with friends, family or faculty, and various mental health problems. Each student experienced multi-dimensional issues, each with their own individual combination of problems, but experienced remediation as a one-dimensional intervention with focus only on improving performance in written exams. What these students needed to be included was help with clinical skills, plus social and emotional support. Fear of termination of the their course was a barrier to open communication with staff. Conclusions These students’ experience of failure was complex. The experience of remediation is influenced by the way in which students make sense of failing. Generic remediation programmes may fail to meet the needs of students for whom personal, social and mental health issues are a part of the picture.

High Stakes and No Takers: The Impact of Florida Comprehensive Assessment Test (FCAT) Writing on Students’ and Teachers’ Perceptions of Writing

This ethnographic case study of ten students and their teachers concludes that the state writing test had a negative impact on students’ and teachers’ perceptions of writing in four categories: strengths and weaknesses in student writing, self-assessment of writing skills, factors impacting test scores, and motivation and attitudes toward writing.

The impact of high stakes testing: the Australian story

High stakes testing in Australia was introduced in 2008 by way of the National Assessment Program – Literacy and Numeracy (NAPLAN). Currently, every year all students in Years 3, 5, 7 and 9 are assessed on the same days using national tests in Reading, Writing, Language Conventions (Spelling, Grammar and Punctuation) and Numeracy. In 2010 the NAPLAN results were published on the Federal Government MySchool website. The impact of these high stakes tests on jurisdictions, school principals, parents and students is considered in this article. We draw on reported observations from the Australian Primary Principals Association during 2009–10 testing periods across the country and published Australian research on the impact of high stakes literacy and numeracy testing. We also examine alternative approaches that include the use of assessment evidence for learning improvement purposes and for accountability purposes. In considering alternatives to the current large-scale testing approach we draw on key insights from research on teacher judgement, achievement standards and social moderation in the context of national curriculum and assessment reform in support of the suggested directions forward.

Dilemmatic spaces : high-stakes testing and the possibilities of collaborative knowledge work to generate learning innovations

This paper examines collaborative researcher-practitioner knowledge work around assessment data in culturally diverse, low- socioeconomic school communities in Queensland, Australia. Specifically, the paper draws on interview accounts about the work of a bridging knowledge flows between a local university and a cluster of schools. We draw on Bernstein’s (2000) concept of recontextualisation to explore the processes of knowledge mediation in dialogues around student assessment data to design instructional innovations. We argue that critical policy studies need to explore the complex ways in which neoliberal education policies are enacted in local sites. Moreover, we suggest that an analysis of collaborative knowledge work designed to improve student learning outcomes in low-socioeconomic school communities necessitates attention to the principles regulating knowledge flows across boundaries. In addition, it necessitates attention to the ways in which mediators navigate dilemmatic spaces, anxieties and affects/feelings in order to generate innovative learning designs in the current global context of high-stakes national testing and accountability regimes.

An ethical approach to sharing school successes in times of high-stakes accountability

High-stakes testing has become an important element of the Australian educational landscape. As one part of the neo-liberal paradigm where beliefs in the individual and the free market are paramount, it is of concern how school leaders can respond to this phenomenon in an ethical manner. Ethics and ethical leadership have increased in prominence both in the educational administration literature and in the media (Cranston, Ehrich, & Kimber, 2006). In this paper we consider ethical theories on which school principals can draw, not only in the leadership of their own schools but in their relationships with other schools. We provide an example of a school leader sharing a successful intervention with other schools, illustrating that school leaders can create spaces for promoting the public good within the context of high-stakes testing.

Approaches to Ensure Quality Standards for Standardized/ Simulated Patient Performance in High Stakes Objective Structured Clinical Exams (OSCEs)

Introduction To meet the quality standards for high-stakes OSCEs, it is necessary to ensure high quality standardized performance of the SPs involved.[1] One of the ways this can be assured is through the assessment of the quality of SPs` performance in training and during the assessment. There is some literature concerning validated instruments that have been used to assess SP performance in formative contexts but very little related to high stakes contexts.[2], [3], [4]. Content and structure During this workshop different approaches to quality control for SPs` performance, developed in medicine, pharmacy and nursing OSCEs, will be introduced. Participants will have the opportunity to use these approaches in simulated interactions. Advantages and disadvantages of these approaches will be discussed. Anticipated outcomes By the end of this session, participants will be able to discuss the rationale for quality control of SPs` performance in high stakes OSCEs, outline key factors in creating strategies for quality control, identify various strategies for assuring quality control, and reflect on applications to their own practice. Who should attend The workshop is designed for those interested in quality assurance of SP performance in high stakes OSCEs. Level All levels are welcome. References Adamo G. 2003. Simulated and standardized patients in OSCEs: achievements and challenges:1992-2003. Med Teach. 25(3), 262- 270. Wind LA, Van Dalen J, Muijtjens AM, Rethans JJ. Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Med Educ 2004, 38(1):39-44. Bouter S, van Weel-Baumgarten E, Bolhuis S. Construction and validation of the Nijmegen Evaluation of the Simulated Patient (NESP): Assessing Simulated Patients' ability to role-play and provide feedback to students. Acad Med: Journal of the Association of American Medical Colleges 2012. May W, Fisher D, Souder D: Development of an instrument to measure the quality of standardized/simulated patient verbal feedback. Med Educ 2012, 2(1).

Quality Control for SP Performance Regarding Patient Portrayal in a High-Stakes OSCE

Introduction Since the quality of patient portrayal of standardized patients (SPs) during an Objective Structured Clinical Exam (OSCE) has a major impact on the reliability and validity of the exam, quality control should be initiated. Literature about quality control of SP’s performance focuses on feedback [1, 2] or completion of checklists [3, 4]. Since we did not find a published instrument meeting our needs for the assessment of patient portrayal, we developed such an instrument after being inspired by others [5] and used it in our high-stakes exam. Methods SP trainers from all five Swiss medical faculties collected and prioritized quality criteria for patient portrayal. Items were revised with the partners twice, based on experiences during OSCEs. The final instrument contains 14 criteria for acting (i.e. adequate verbal and non-verbal expression) and standardization (i.e. verbatim delivery of the first sentence). All partners used the instrument during a high-stakes OSCE. Both, SPs and trainers were introduced to the instrument. The tool was used in training (more than 100 observations) and during the exam (more than 250 observations). FAIR_OSCE The list of items to assess the quality of the simulation by SPs was primarily developed and used to provide formative feedback to the SPs in order to help them to improve their performance. It was therefore named “Feedbackstruckture for the Assessment of Interactive Role play in Objective Structured Clinical Exams (FAIR_OSCE). It was also used to assess the quality of patient portrayal during the exam. The results were calculated for each of the five faculties individually. Formative evaluation was given to the five faculties with individual feedback without revealing results of other faculties other than overall results. Results High quality of patient portrayal during the exam was documented. More than 90% of SP performances were rated to be completely correct or sufficient. An increase in quality of performance between training and exam was noted. In example the rate of completely correct reaction in medical tests increased from 88% to 95%. 95% completely correct reactions together with 4% sufficient reactions add up to 99% of the reactions meeting the requirements of the exam. SP educators using the instrument reported an augmentation of SPs performance induced by the use of the instrument. Disadvantages mentioned were high concentration needed to explicitly observe all criteria and cumbersome handling of the paper-based forms. Conclusion We were able to document a very high quality of SP performance in our exam. The data also indicate that our training is effective. We believe that the high concentration needed using the instrument is well invested, considering the observed augmentation of performance. The development of an iPad based application for the form is planned to address the cumbersome handling of the paper.

How Can Teachers Alleviate Parental Anxiety in High Stakes Testing?

In Florida, third and tenth graders are required to take the Florida Comprehensive Assessment Test (FCAT), a high stakes test. Third graders must score a Level 2 or higher on the Sunshine States Standards portion of the FCAT or fifty-one percent or higher on the Normreference portion of the FCAT in order to be promoted. In 2003, the Florida Department of Education reported 31 percent of third graders in Miami-Dade County were retained (2003). The purpose of the study is to investigate how third grade teachers can decrease the level of parental anxiety for this high stakes test and determine if there are different levels of parental anxiety based upon certain variables. Parent workshops will be offered to families focusing on FCAT strategies and relaxation techniques designed to increase parental efficacy and decrease parental anxiety related to high stakes testing.

Retention and special education referral practices before and after the implementation of high -stakes testing

The rewards and sanctions associated with high-stakes testing may induce educators to participate in practices that will ensure the elimination of the scores of low-achieving students from the testing pool. Two ways in which scores may be eliminated is through retention or referral to special education. ^ This study examined the use of these practices at 179 elementary schools in Miami-Dade County Public Schools, the 4th largest school district in the country. Between- and within-subjects designs were analyzed using repeated measures analysis of variance to compare retention and referral to special education practices over a five-year period of time, two years prior to and two years after the implementation of Florida's high-stakes test, the Florida Comprehensive Assessment Test, FCAT. ^ Significant main effects for referral and retention over time were demonstrated. The use of retention steadily increased over the first three years, with its usage maintained during the fourth year. While the use of referral actually decreased from the first to second years, a significant change occurred after the implementation of the FCAT. ^ Examination of the use of these practices according to student and school characteristics revealed significant differences. Increases in the use of referral across time was significant for Black, non-Hispanic and Hispanic students, all limited English proficiency population categories, medium and low socioeconomic status category schools, all grade levels, and for schools with accountability grades of A. C, D and F with the most striking absolute increase occurring for F schools. Increases in the use of retention across time were significant for all ethnic groups, limited English proficiency categories, and socioeconomic status categories, for grades kindergarten through four and by gender. Significant increases occurred for schools with accountability performance grades of C, D and F; however the most dramatic increase occurred for the F schools. A direct relationship between performance category grade of school and their use of retention was demonstrated. The results suggest that schools changed their use of referral and retention in response to the implementation of the FCAT. ^

Retention and special education referral practices before and after the implementation of high-stakes testing

The rewards and sanctions associated with high-stakes testing may induce educators to participate in practices that will ensure the elimination of the scores of low-achieving students from the testing pool. Two ways in which scores may be eliminated is through retention or referral to special education. This study examined the use of these practices at 179 elementary schools in Miami-Dade County Public Schools, the 4th largest school district in the country. Between- and within-subjects designs were analyzed using repeated measures analysis of variance to compare retention and referral to special education practices over a five-year period of time, two years prior to and two years after the implementation of Florida's high-stakes test, the Florida Comprehensive Assessment Test, FCAT. Significant main effects for referral and retention over time were demonstrated. The use of retention steadily increased over the first three years, with its usage maintained during the fourth year. While the use of referral actually decreased from the first to second years, a significant change occurred after the implementation of the FCAT. Examination of the use of these practices according to student and school characteristics revealed significant differences. Increases in the use of referral across time was significant for Black, non-Hispanic and Hispanic students, all limited English proficiency population categories, medium and low socioeconomic status category schools, all grade levels, and for schools with accountability grades of A, C, D and F with the most striking absolute increase occurring for F schools. Increases in the use of retention across time were significant for all ethnic groups, limited English proficiency categories, and socioeconomic status categories, for grades kindergarten through four and by gender. Significant increases occurred for schools with accountability performance grades of C, D and F; however the most dramatic increase occurred for the F schools. A direct relationship between performance category grade of school and their use of retention was demonstrated. The results suggest that schools changed their use of referral and retention in response to the implementation of the FCAT.

UNDERSTANDING ESSAY RATING AS A SOCIALLY MEDIATED ACTIVITY: THE CASE OF A HIGH-STAKES ENGLISH TEST IN CHINA

Most essay rating research in language assessment has examined human raters’ essay rating as a cognitive process, thus overlooking or oversimplifying the interaction between raters and sociocultural contexts. Given that raters are social beings, their practices have social meanings and consequences. Hence it is important to situate essay rating within its sociocultural context for a more meaningful understanding. Drawing on Engeström’s (1987, 2001) cultural-historical activity theory (CHAT) framework with a sociocultural perspective, this study reconceptualized essay rating as a socially mediated activity with both cognitive (individual raters’ goal-directed decision-making actions) and social layers (raters’ collective object-oriented essay rating activity at related settings). In particular, this study explored raters’ essay rating at one provincial rating centre in China within the context of a high-stakes university entrance examination, the National Matriculation English Test (NMET). This study adopted a multiple-method multiple-perspective qualitative case study design. Think-aloud protocols, stimulated recalls, interviews, and documents served as the data sources. This investigation involved 25 participants at two settings (rating centre and high schools), including rating centre directors, team leaders, NMET essay raters who were high school teachers, and school principals and teaching colleagues of these essay raters. Data were analyzed using Strauss and Corbin’s (1990) open and axial coding techniques, and CHAT for data integration. The findings revealed the interaction between raters and the NMET sociocultural context. Such interaction can be understood through a surface structure (cognitive layer) and a deep structure (social layer) concerning how raters assessed NMET essays, where the surface structure reflected the “what” and the deep structure explained the “how” and “why” in raters’ decision-making. This study highlighted the roles of goals and rules in rater decision-making, rating tensions and raters’ solutions, and the relationship between essay rating and teaching. This study highlights the value of a sociocultural view to essay rating research, demonstrates CHAT as a sociocultural approach to investigate essay rating, and proposes a direction for future washback research on the effect of essay rating. This study also provides support for NMET rating practices that can potentially bring positive washback to English teaching in Chinese high schools.

Does gender make a difference? Gender-related fairness of high-stakes testing in A-level examinations in English as foreign language in the German state of North Rhine-Westphalia in the context of Educational Governance

The shift from decentralized to centralized A-level examinations (Abitur) was implemented in the German school system as a measure of Educational Governance in the last decade. This reform was mainly introduced with the intention of providing higher comparability of school examinations and student achievement as well as increasing fairness in school examinations. It is not known yet if these ambitious aims and functions of the new centralized examination format have been achieved and if fairer assessment can be guaranteed in terms of providing all students with the same opportunities to pass the examinations by allocating fair tests to different student subpopulations e.g., students of different background or gender. The research presented in this article deals with these questions and focuses on gender differences. It investigates gender-specific fairness of the test items in centralized Abitur examinations as high school exit examinations in Germany. The data are drawn from Abitur examinations in English (as a foreign language). Differential item functioning (DIF) analysis reveals that at least some parts of the examinations indicate gender inequality. (DIPF/Orig.)