The Comprehensive Assessment of Leadership for Learning: A Next Generation Formative Evaluation and Feedback System

Recent developments in federal policy have prompted the creation of state evaluation frameworks for principals and teachers that hold educators accountable for effective practices and student outcomes. These changes have created a demand for formative evaluation instruments that reflect current accountability pressures and can be used by schools to focus school improvement and leadership development efforts. The Comprehensive Assessment of Leadership for Learning (CALL) is a next generation, 360-degree on-line assessment and feedback system that reflect best practices in feedback design. Some unique characteristics of CALL include a focus on: leadership distributed throughout the school rather than as carried out by an individual leader; assessment of leadership tasks rather than perceptions of leadership practice; a focus on larger complex systems of middle and high school; and transparency of assessment design. This paper describes research contributing to the design and validation of the CALL survey instrument.

Harnessing assessment and feedback in the first year to support learning success, engagement and retention

It has been argued that intentional first year curriculum design has a critical role to play in enhancing first year student engagement, success and retention (Kift, 2008). A fundamental first year curriculum objective should be to assist students to make the successful transition to assessment in higher education. Scott (2006) has identified that ‘relevant, consistent and integrated assessment … [with] prompt and constructive feedback’ are particularly relevant to student retention generally; while Nicol (2007) suggests that ‘lack of clarity regarding expectations in the first year, low levels of teacher feedback and poor motivation’ are key issues in the first year. At the very minimum, if we expect first year students to become independent and self-managing learners, they need to be supported in their early development and acquisition of tertiary assessment literacies (Orrell, 2005). Critical to this attainment is the necessity to alleviate early anxieties around assessment information, instructions, guidance, and performance. This includes, for example:  inducting students thoroughly into the academic languages and assessment genres they will encounter as the vehicles for evidencing learning success; and  making expectations about the quality of this evidence clear. Most importantly, students should receive regular formative feedback of their work early in their program of study to aid their learning and to provide information to both students and teachers on progress and achievement. Leveraging research conducted under an ALTC Senior Fellowship that has sought to articulate a research-based 'transition pedagogy' (Kift & Nelson, 2005) – a guiding philosophy for intentional first year curriculum design and support that carefully scaffolds and mediates the first year learning experience for contemporary heterogeneous cohorts – this paper will discuss theoretical and practical strategies and examples that should be of assistance in implementing good assessment and feedback practices across a range of disciplines in the first year.

How do the learning outcomes of students compare in units with different levels of practical work? A comparative study about the relation between Problem-­Based Learning and assessment approaches

The literature supporting the notion that active, student-centered learning is superior to passive, teacher-centered instruction is encyclopedic (Bonwell & Eison, 1991; Bruning, Schraw, & Ronning, 1999; Haile, 1997a, 1997b, 1998; Johnson, Johnson, & Smith, 1999). Previous action research demonstrated that introducing a learning activity in class improved the learning outcomes of students (Mejias, 2010). People acquire knowledge and skills through practice and reflection, not by watching and listening to others telling them how to do something. In this context, this project aims to find more insights about the level of interactivity in the curriculum a class should have and its alignment with assessment so the intended learning outcomes (ILOs) are achieved. In this project, interactivity is implemented in the form of problem- based learning (PBL). I present the argument that a more continuous formative feedback when implemented with the correct amount of PBL stimulates student engagement bringing enormous benefits to student learning. Different levels of practical work (PBL) were implemented together with two different assessment approaches in two subjects. The outcomes were measured using qualitative and quantitative data to evaluate the levels of student engagement and satisfaction in the terms of ILOs.

Development and implementation of the Australian universities radiation therapy student clinical assessment form

Purpose: Prior to 2009, one of the problems faced by radiation therapists who supervised and assessed students on placement in Australian clinical centres, was that each of the six Australian universities where Radiation Therapy (RT) programmes were conducted used different clinical assessment and reporting criteria. This paper describes the development of a unified national clinical assessment and reporting form that was implemented nationally by all six universities in 2009. Methods: A four phase methodology was used to develop the new assessment form and user guide. Phase 1 included university consensus around domains of student practice and assessment, and alignment with national competency standards; Phase 2 was a national consensus workshop attended by radiation therapists involved in student supervision and assessment; Phase 3 was an action research re-iterative Delphi technique involving two rounds of a mail-out to gain further expert consensus; and stage 4 was national piloting of the developed assessment form. Results: The new assessment form includes five main domains of practice and 19 sub-domain criteria which students are assessed against during placement. Feedback from the pilot centre participants was positive, with the new form being assessed to be comprehensive and complemented by the accompanying user guide. Conclusion: The new assessment form has improved both the formative and summative assessment of students on placement, as well as enhancing the quality of feedback to students and the universities. The new national form has high acceptance from the Australian universities and has been subject to wide review by the profession.

Towards fairer assessment

Drawing on the largest Australian collection and analysis of empirical data on multiple facets of Aboriginal and Torres Strait Islander education in state schools to date, this article critically analyses the systemic push for standardized testing and improved scores, and argues for a greater balance of assessment types by providing alternative, inclusive, participatory approaches to student assessment. The evidence for this article derives from a major evaluation of the Stronger Smarter Learning Communities. The first large-scale picture of what is occurring in classroom assessment and pedagogy for Indigenous students is reported in this evaluation yet the focus in this article remains on the issue of fairness in student assessment. The argument presented calls for “a good balance between formative and summative assessment” (OECD, Synergies for Better Learning An International Perspective on Evaluation and Assessment, Pointers for Policy Development, 2013) at a time of unrelenting high-stakes, standardized testing in Australia with a dominance of secondary as opposed to primary uses of NAPLAN data by systems, schools and principals. A case for more “intelligent accountability in education” (O’Neill, Oxford Review of Education 39(1):4–16, 2013) together with a framework for analyzing efforts toward social justice in education (Cazden, International Journal of Educational Psychology 1(3):178–198, 2012) and fairer assessment make the case for more alternative assessment practices in recognition of the need for teachers’ pedagogic practice to cater for increased diversity.

OSCE online : making assessment of clinical competencies transparent, accountable & authentic

Creating an authentic assessment which at once assesses competencies, scene management, communication and overall patient care is challenging in the competitive tertiary education market. Increasing student numbers and the cost of evaluating scenario based competencies serve to ensure the need for consistent objectivity and need for timely feedback to students on their performance. Objective structured clinical examination (OSCE) is currently the most flexible approach to competency based formative and summative assessment and widely used within paramedic degree programs. Students are understandably compelled to perform well and can be frustrated by not receiving timely and appropriate feedback. Increasingly a number of products aimed at providing a more efficient and paperless approach have begun to enter the market. These products, it is suggested are aimed at medicine programs and not at allied health professions and limited to one operating system and therefore ignore issues surrounding equity and accessibility. OSCE Online aims to address this gap in the market and is tailored to these disciplines. The application will provide a service that can be both tailored and standardised from a pre-written bank, depending upon requirement to fit around the needs of clinical competency assessment. Delivering authentic assessments to address student milestones in their training to become paramedics is the cornerstone of OSCE Online. By not being restricted to a specific device it will address issues of functionality, adaptability, accessibility, authenticity and importantly: transparency and accountability by producing contemporaneous data allowing issues to be easily identified and rectified.

The effect of feedback from pupil to teacher on assessment for leaning and visible learning: An ethnographic case study in a community school in England and the outcome in a state high school in Queensland, Australia

This study shows that there is positive regulatory effect of feedback from pupils to teachers on Assessment for Learning (AfL), classroom proactiveness, and on visible and progressive learning but not on behaviour. This research finding further articulates feedback from pupil to teacher as a paradigm shift from the classical paradigm of feedback from teacher to pupil. Here, the emphasis is geared towards pupils understanding of objectives built from previous knowledge. These are then feedback onto the teachers by the pupils in the form of discrete loops of cues and questions, where they are with their learning. This therefore enables them to move to the next level of understanding, and thus acquired independence, which in turn is reflected by their success in both formative and summative assessments. This study therefore shows that when feedback from pupil to teacher is used in combination with teacher to pupil feedback, AfL is ameliorated and hence, visible and accelerated learning occurs in a gender, nor subject non-dependent manner.

In-training assessment - Its potential in enhancing clinical teaching

Context In-training assessment (ITA) has established its place alongside formative and summative assessment at both the undergraduate and postgraduate level. In this paper the authors aimed to identify those characteristics of ITA that could enhance clinical teaching. Methods A literature review and discussions by an expert working group at the Ninth Cambridge Conference identified the aspects of ITA that could enhance clinical teaching. Results The features of ITA identified included defining the specific benefits to the learner, teacher and institution, and highlighting the patient as the context for ITA and clinical teaching. The ‘mapping’ of a learner’s progress towards the clinical teaching objectives by using multiple assessments over time, by multiple observers in both a systematic and opportunistic way correlates with the incremental nature of reaching clinical competence. Conclusions The importance of ITA based on both direct and indirect evidence of what the learner actually does in the real clinical setting is emphasized. Particular attention is given to addressing concerns in the more controversial areas of assessor training, ratings and documentation for ITA. Areas for future research are also identified.

Time Efficient Assessment and Feedback Methods for Large Computer-Aided-Design Cohorts

Timely and individualized feedback on coursework is desirable from a student perspective as it facilitates formative development and encourages reflective learning practice. Faculty however are faced with a significant and potentially time consuming challenge when teaching larger cohorts if they are to provide feedback which is timely, individualized and detailed. Additionally, for subjects which assess non-traditional submissions, such as Computer-Aided-Design (CAD), the methods for assessment and feedback tend not to be so well developed or optimized. Issues can also arise over the consistency of the feedback provided. Evaluations of Computer-Assisted feedback in other disciplines (Denton et al, 2008), (Croft et al, 2001) have shown students prefer this method of feedback to traditional “red pen” marking and also that such methods can be more time efficient for faculty.
Herein, approaches are described which make use of technology and additional software tools to speed up, simplify and automate assessment and the provision of feedback for large cohorts of first and second year engineering students studying modules where CAD files are submitted electronically. A range of automated methods are described and compared with more “manual” approaches. Specifically one method uses an application programming interface (API) to interrogate SolidWorks models and extract information into an Excel spreadsheet, which is then used to automatically send feedback emails. Another method describes the use of audio recordings made during model interrogation which reduces the amount of time while increasing the level of detail provided as feedback.
Limitations found with these methods and problems encountered are discussed along with a quantified assessment of time saving efficiencies made.

Improving E-Assessment Systems With A focus On Adaptive Feedback And Marking

Institutions involved in the provision of tertiary education across Europe are feeling the pinch. European universities, and other higher education (HE) institutions, must operate in a climate where the pressure of government spending cuts (Garben, 2012) is in stark juxtaposition to the EU’s strategy to drive forward and maintain a growth of student numbers in the sector (eurostat, 2015).

In order to remain competitive, universities and HE institutions are making ever-greater use of electronic assessment (E-Assessment) systems (Chatzigavriil et all, 2015; Ferrell, 2012). These systems are attractive primarily because they offer a cost-effect and scalable approach for assessment. In addition to scalability, they also offer reliability, consistency and impartiality; furthermore, from the perspective of a student they are most popular because they can offer instant feedback (Walet, 2012).

There are disadvantages, though.

First, feedback is often returned to a student immediately on competition of their assessment. While it is possible to disable the instant feedback option (this is often the case during an end of semester exam period when assessment scores must be can be ratified before release), however, this option tends to be a global ‘all on’ or ‘all off’ configuration option which is controlled centrally rather than configurable on a per-assessment basis.

If a formative in-term assessment is to be taken by multiple groups of
students, each at different times, this restriction means that answers to each question will be disclosed to the first group of students undertaking the assessment. As soon as the answers are released “into the wild” the academic integrity of the assessment is lost for subsequent student groups.

Second, the style of feedback provided to a student for each question is often limited to a simple ‘correct’ or ‘incorrect’ indicator. While this type of feedback has its place, it often does not provide a student with enough insight to improve their understanding of a topic that they did not answer correctly.

Most E-Assessment systems boast a wide range of question types including Multiple Choice, Multiple Response, Free Text Entry/Text Matching and Numerical questions. The design of these types of questions is often quite restrictive and formulaic, which has a knock-on effect on the quality of feedback that can be provided in each case.

Multiple Choice Questions (MCQs) are most prevalent as they are the most prescriptive and therefore most the straightforward to mark consistently. They are also the most amenable question types, which allow easy provision of meaningful, relevant feedback to each possible outcome chosen.
Text matching questions tend to be more problematic due to their free text entry nature. Common misspellings or case-sensitivity errors can often be accounted for by the software but they are by no means fool proof, as it is very difficult to predict in advance the range of possible variations on an answer that would be considered worthy of marks by a manual marker of a paper based equivalent of the same question.

Numerical questions are similarly restricted. An answer can be checked for accuracy or whether it is within a certain range of the correct answer, but unless it is a special purpose-built mathematical E-Assessment system the system is unlikely to have computational capability and so cannot, for example, account for “method marks” which are commonly awarded in paper-based marking.

From a pedagogical perspective, the importance of providing useful formative feedback to students at a point in their learning when they can benefit from the feedback and put it to use must not be understated (Grieve et all, 2015; Ferrell, 2012).

In this work, we propose a number of software-based solutions, which will overcome the limitations and inflexibilities of existing E-Assessment systems.

Assessment and learning : is it time to rethink student activities and academic roles?

Resumen basado en el de la publicación

How Student Self-Assessment Influences Mastery Of Objectives

In this action research study of my classroom of 7th grade students, enrolled in Pre- Algebra (an 8th grade course), I investigated: rate of homework completion when not taken as part of the academic grade, cognizant self-assessment and its affect on mastery of objectives, and use of self-assessment to guide instruction and re-teaching of classroom objectives. I learned that without sufficient accountability homework completion rates drop with time. Similarly, students can be overconfident in their abilities but unmoved when their summative reports do not match their initial perceived formative benchmarks. Finally, due in part to our society’s reactive nature; students find it more practical to play catch-up rather than staying caught up. As a result of this research, I plan to create, with the help of the students, an accountability statute to help students stay caught up with their understanding of the objectives, as well as allow additional time and energy spent by both student and teacher to react in a timely manner to complete student knowledge within a day or two rather than a week or two later.

Formatives Selbst- und Peer-Assessment im Unterricht peripher venöser Venenpunktion

Das Lernen einer Fertigkeit durch Demonstration und anschließendes Üben wird „Modeling“ genannt. Es basiert darauf, die Diskrepanz zwischen dem Soll-Zustand (Demonstration) und dem Ist-Zustand (Üben) zu erkennen und zu beheben. Dafür ist die exakte Analyse der eigenen Fertigkeiten beim Üben unentbehrlich. Entsprechend ist auch bekannt, dass formative Evaluationen wesentlich zum erfolgreichen Lernen beitragen. Wir haben deshalb im Kurs für periphere Venenpunktion im 3. Studienjahr formatives Selbst- und Peer-Assessment eingeführt. Die Struktur des Assessment entspricht einem DOPS (dircect observation of procedural skills). DOPS stammt aus dem Arbeitsplatz-basieren Assessment und beinhaltet die Beurteilung folgender Kriterien: Vorbereitung/Nachsorge, technische Fertigkeit, Asepsis/Sicherheit, klinische Urteilsfähigkeit, Organisation/Effizienz, professionelles Verhalten, Gesamteindruck. Diese Kriterien wurden für den Unterricht konkretisiert (z.B. Vorbereitung mit Beschriftung der Röhrchen, etc.) und den Studierenden als Merkblätter ausgeteilt. Die Studierenden beurteilten ihre eigene Performance bzw. die eines Kommilitonen, gaben sich Feedback und legten individuelle Lernziele zur Verbesserung fest. Dieses Vorgehen hat den Vorteil, dass sowohl der Übende, als auch der beobachtende Kommilitone, die optimale Ausführung der jeweiligen Tätigkeit reflektieren, welches für beide eine Möglichkeit zum Lernen bietet . Bei der Evaluation des Kurses wurden die Handouts mit den Kriterien der DOPS von Teilnehmern von 9 der 10 Gruppen positiv erwähnt. Im Rahmen eines Debriefing mit den studentischen Tutoren wurde jedoch kritisch angemerkt, dass der Prozess der formativen Selbst- und Fremdevaluation den Studierenden im 3. Studienjahr nicht vertraut war. Es war für die Teilnehmer schwierig konkretes Feedback zu geben und individuelle Lernziele festzulegen. Für das kommende Jahr planen wir in Bezug auf den Kurs folgendes: Die Kriterien der korrekten Durchführung einer Fertigkeit zu formulieren wird von den Teilnehmern als hilfreich empfunden und soll deshalb beibehalten werden. Die Studierenden, die dieses Jahr an dem Kurs teilnehmen, haben bereits ein Feedbacktraining absolviert. Der Kurs kann deshalb neu an Vorkenntnisse anknüpfen. Darüber hinaus soll der Prozess der Festlegung der individuellen Lernziele in der Schulung der studentischen Tutoren des Kurses mehr Gewicht erhalten, damit die Tutoren die Teilnehmer hier gezielt unterstützen können.