A First Step Towards Synthesising Rubrics and Video for the Formative Assessment of Complex Skills

Abstract. The performance objectives used for the formative assessment of com- plex skills are generally set through text-based analytic rubrics[1]. Moreover, video modeling examples are a widely applied method of observational learning, providing students with context-rich modeling examples of complex skills that act as an analogy for problem solving [1]. The purpose of this theoretical paper is to synthesize the components of video modeling and rubrics to support the formative assessment of complex skills. Based on theory, we argue that application of the developed Video Enhanced Rubrics (VER) fosters learners’ development of mental models, quality of provided feedback by various actors and finally, the learners mastery of complex skills.

Viewbrics: Formative Assessment of Complex Skills with Video-Enhanced Rubrics (VER) in Dutch Secondary Education

To learn complex skills, like collaboration, learners need to acquire a concrete and consistent mental model of what it means to master this skill. If learners know their current mastery level and know their targeted mastery level, they can better determine their subsequent learning activities. Rubrics support learners in judging their skill performance as they provide textual descriptions of skills’ mastery levels with performance indicators for all constituent subskills. However, text-based rubrics have a limited capacity to support the formation of mental models with contextualized, time-related and observable behavioral aspects of a complex skill. This paper outlines the design of a study that intends to investigate the effect of rubrics with video modelling examples compared to text-based rubrics on skills acquisition and feedback provisioning. The hypothesis is that video-enhanced rubrics, compared to text based rubrics, will improve mental model formation of a complex skill and improve the feedback quality a learner receives (from e.g. teachers, peers) while practicing a skill, hence positively effecting final mastery of a skill.

A First Step Towards Video Enhanced Rubrics for the Formative Assessment of Complex Skills : A literature review and a plea

Paper presentation at the TEA2016 conference, Tallinn, Estonia.

A preliminary study of open learner model representation formats to support formative assessment

Open learner models provide a way of showing teachers and students the learning progress of a student against expectations. Creating an effective interface to present a learner model is an important part in open learner modeling. Usefulness of the learner model relies on using a clear andeffective representation format to facilitate users’understanding of the information presented. In this paper, wepropose a range of open learner model representation formatsto display students’ learning task status and achievement interms of learning outcomes. We investigate different types ofrepresentation formats and data useful for users to inspect thelearner model. An interface design prototype has been built toprovide potential users with an evaluation platform thatenables investigation of these aspects. The results obtained willallow us to develop a more effective new open learner modelvisualisation tool.

Assessment and curriculum reform in China : the college English test and tertiary English as a foreign language education.

Educational assessment was a worldwide commonplace practice in the last century. With the theoretical underpinnings of education shifting from behaviourism and social efficiency to constructivism and cognitive theories in the past two decades, the assessment theories and practices show a widespread changing movement. The emergent assessment paradigm, with a futurist perspective, indicates a deviation away from the prevailing large scale high-stakes standardised testing and an inclination towards classroom-based formative assessment. Innovations and reforms initiated in attempts to achieve better education outcomes for a sustainable future via more developed learning and assessment theories have included the 2007 College English Reform Program (CERP) in Chinese higher education context. This paper focuses on the College English Test (CET) - the national English as a Foreign Language (EFL) testing system for non-English majors at tertiary level in China. It seeks to explore the roles that the CET played in the past two College English curriculum reforms, and the new role that testing and assessment assumed in the newly launched reform. The paper holds that the CET was operationalised to uplift the standards. However, the extended use of this standardised testing system brings constraints as well as negative washback effects on the tertiary EFL education. Therefore in the newly launched reform -CERP, a new assessment model which combines summative and formative assessment approaches is proposed. The testing and assessment, assumed a new role - to engender desirable education outcomes. The question asked is: will the mixed approach to formative and summative assessment provide the intended cure to the agony that tertiary EFL education in China has long been suffering - spending much time, yet achieving little effects? The paper reports the progresses and challenges as informed by the available research literature, yet asserts a lot needs to be explored on the potential of the assessment mix in this examination tradition deep-rooted and examination-obsessed society.

Towards learner autonomy : an assessment for learning approach

Using Assessment for Learning (AfL) may develop learner autonomy however, very often AfL is reduced to a set of strategies that do not always achieve the desired outcome. This research adopted a different approach that examined AfL as a cultural practice, situated within influential social relationships that shape learner identity. The study addressed the question “What are the qualities of the teacher-student relationship that support student learning autonomy in an AfL context?” Three case studies of the interactions of Queensland middle school teachers and their classes of Year 7, 8 and 9 were developed over one year. Data were collected from field notes and video recordings of classroom interactions and individual and focus group interviews with teachers and students. The analysis began with a close look at the field data. Interpretations that emerged from a sociocultural theoretical understanding were helpful in informing the process of analysis. Themes and patterns of interrelationships were identified through thematic coding using a constant comparative approach. Validation was achieved through methodological triangulation. Four findings that inform an understanding of AfL and the development of learner autonomy emerged. Firstly, autonomy is theorised as a context-specific identity mediated through the teacher-student relationship. Secondly, it was observed that learners negotiated their identities as knowers through AfL practices in various tacit, explicit, group and individual ways in a ‘generative dance’ of knowing in action (Cook & Brown, 2005). Thirdly, teachers and learners negotiated their participation by drawing from identities in multiple communities of practice. Finally it is proposed that a new participative identity or narrative for assessment is needed. This study contributes to understandings about teacher AfL practices that can help build teacher assessment capacity. Importantly, autonomy is understood as an identity that is available to all learners. This study is also significant as it affirms the importance of teacher assessment to support learners in developing autonomy, a focus that challenges the singular assessment policy focus on measuring performance. Finally this study contributes to a sociocultural theoretical understanding of AfL.

Assessment policy change in relation to English language teaching and learning in China : a study of perspectives from two universities

In response to concerns about the quality of English Language Learning (ELL) education at tertiary level, the Chinese Ministry of Education (CMoE) launched the College English Reform Program (CERP) in 2004. By means of a press release (CMoE, 2005) and a guideline document titled College English Curriculum Requirements (CECR) (CMoE, 2007), the CERP proposed two major changes to the College English assessment policy, which were: (1) the shift to optional status for the compulsory external test, the College English Test Band 4 (CET4); and (2) the incorporation of formative assessment into the existing summative assessment framework. This study investigated the interactions between the College English assessment policy change, the theoretical underpinnings, and the assessment practices within two Chinese universities (one Key University and one Non-Key University). It adopted a sociocultural theoretical perspective to examine the implementation process as experienced by local actors of institutional and classroom levels. Systematic data analysis using a constant comparative method (Merriam, 1998) revealed that contextual factors and implementation issues did not lead to significant differences in the two cases. Lack of training in assessment and the sociocultural factors such as the traditional emphasis on the product of learning and hierarchical teacher/students relationship are decisive and responsible for the limited effect of the reform.

Using neo-Piagetian theory, formative in-Class tests and think alouds to better understand student thinking : a preliminary report on computer programming

It is acknowledged around the world that many university students struggle with learning to program (McCracken et al., 2001; McGettrick et al., 2005). In this paper, we describe how we have developed a research programme to systematically study and incrementally improve our teaching. We have adopted a research programme with three elements: (1) a theory that provides an organising framework for defining the type of phenomena and data of interest, (2) data on how the class as a whole performs on formative assessment tasks that are framed from within the organising framework, and (3) data from one-on-one think aloud sessions, to establish why students struggle with some of those in-class formative assessment tasks. We teach introductory computer programming, but this three-element structure of our research is applicable to many areas of engineering education research.

Using formative feedback to identify and support first year chemistry students with missing or misconceptions. A Practice Report

Students entering tertiary studies possess a diverse range of prior experiences in their academic preparation for tertiary chemistry so academics need tools to enable them to respond to issues in diversity in conceptual models possessed by entering students. Concept inventories can be used to provide formative feedback to help students identify concepts that they need to address to improve construction of subsequent understanding enabling their learning. Modular, formative learning activities that can be administered inside or outside of class in first year chemistry courses have been developed. These activities address key missing and mis-conceptions possessed by incoming student. Engagement in these learning activities by students and academics will help shift the culture of diagnostic and formative assessment within the tertiary context and address issues around the secondary/tertiary transition. This diagnostic/intervention framework is currently being trialed across five Australian tertiary institutions encompassing a large heterogeneous sample of students.