A First Step Towards Synthesising Rubrics and Video for the Formative Assessment of Complex Skills

Abstract. The performance objectives used for the formative assessment of com- plex skills are generally set through text-based analytic rubrics[1]. Moreover, video modeling examples are a widely applied method of observational learning, providing students with context-rich modeling examples of complex skills that act as an analogy for problem solving [1]. The purpose of this theoretical paper is to synthesize the components of video modeling and rubrics to support the formative assessment of complex skills. Based on theory, we argue that application of the developed Video Enhanced Rubrics (VER) fosters learners’ development of mental models, quality of provided feedback by various actors and finally, the learners mastery of complex skills.

Viewbrics: Formative Assessment of Complex Skills with Video-Enhanced Rubrics (VER) in Dutch Secondary Education

To learn complex skills, like collaboration, learners need to acquire a concrete and consistent mental model of what it means to master this skill. If learners know their current mastery level and know their targeted mastery level, they can better determine their subsequent learning activities. Rubrics support learners in judging their skill performance as they provide textual descriptions of skills’ mastery levels with performance indicators for all constituent subskills. However, text-based rubrics have a limited capacity to support the formation of mental models with contextualized, time-related and observable behavioral aspects of a complex skill. This paper outlines the design of a study that intends to investigate the effect of rubrics with video modelling examples compared to text-based rubrics on skills acquisition and feedback provisioning. The hypothesis is that video-enhanced rubrics, compared to text based rubrics, will improve mental model formation of a complex skill and improve the feedback quality a learner receives (from e.g. teachers, peers) while practicing a skill, hence positively effecting final mastery of a skill.

A First Step Towards Video Enhanced Rubrics for the Formative Assessment of Complex Skills : A literature review and a plea

Paper presentation at the TEA2016 conference, Tallinn, Estonia.

Pratiche di formative assessment per promuovere abilita di comprensione dei testi. Una sperimentazione in una scuola secondaria di primo grado

I dati delle indagini sugli apprendimenti degli studenti in Italia rivelano l’esistenza di fragilità nell’acquisizione di competenze essenziali e di differenze tra i risultati conseguiti. Per innovare la didattica al fine di adeguarla ai bisogni degli studenti, gli esperti di Docimologia caldeggiano l’uso di pratiche di valutazione formativa, o formative assessment (FA). In ambito internazionale diversi studi hanno mostrato l'efficacia di tali prassi, mentre in Italia non esistono ricerche sperimentali finalizzate a studiarne l’impatto sugli apprendimenti. Il progetto è entrato all’interno di quest’ambito di studi per controllare l’efficacia di un insieme di pratiche di FA sull'incremento delle abilità di comprensione dei testi degli studenti. Lo scopo è stato perseguito realizzando una sperimentazione in una scuola secondaria di primo grado che ha coinvolto gli studenti di due classi prime, i quali sono stati suddivisi a metà attraverso tecniche di randomizzazione per formare i due gruppi, sperimentale e di controllo. Dopo aver effettuato una rilevazione iniziale delle abilità di comprensione dei testi degli studenti (pre-test), è stato realizzato con quelli del gruppo sperimentale un intervento composto da 15 incontri di FA della durata di due ore ciascuno. Alla fine, sono state effettuate due rilevazioni finali (post-test) utilizzando sia la stessa prova utilizzata come pre-test sia una prova parallela. È stata calcolata la differenza post-test-pre-test per ogni gruppo ed è stato verificato quanto avesse influito la partecipazione all’intervento sperimentale su tale differenza tramite test non parametrici. I risultati hanno mostrato un incremento di abilità lievemente maggiore nel gruppo sperimentale, se confrontato con quello del gruppo di controllo, anche se questa differenza tra i due gruppi non è statisticamente significativa. Sebbene le analisi non abbiano consentito di rifiutare l’ipotesi nulla, la rilevanza di tale progetto risiede nel tentativo di aprire il dibattito sull’efficacia di prassi di FA sugli apprendimenti degli studenti in Italia.

Development, implementation, and assessment of a distance module in endocrine physiology

Rangel EM, Mendes IA, Carnio EC, Marchi Alves LM, Godoy S, Crispim JA. Development, implementation, and assessment of a distance module in endocrine physiology. Adv Physiol Educ 34: 70-74, 2010; doi: 10.1152/advan.00070.2009.-This study aimed to develop, implement, and assess a distance module in endocrine physiology in TelEduc for undergraduate nursing students from a public university in Brazil, with a sample size of 44 students. Stage 1 consisted of the development of the module, through the process of creating a distance course by means of the Web. Stage 2 was the planning of the module's practical functioning, and stage 3 was the planning of student evaluations. In the experts' assessment, the module complied with pedagogical and technical requirements most of the time. In the practical functioning stage, 10 h were dedicated for on-site activities and 10 h for distance activities. Most students (93.2%) were women between 19 and 23 yr of age (75%). The internet was the most used means to remain updated for 23 students (59.0%), and 30 students (68.2%) accessed it from the teaching institution. A personal computer was used by 23 students (56.1%), and most of them (58.1%) learned to use it alone. Access to a forum was more dispersed (variation coefficient: 86.80%) than access to chat (variation coefficient: 65.14%). Average participation was 30 students in forums and 22 students in the chat. Students' final grades in the module averaged 8.5 (SD: 1.2). TelEduc was shown to be efficient in supporting the teaching- learning process of endocrine physiology.

E-assessment in mathematics courses with multiple-choice questions tests

With the implementation of the Bologna Process several challenges have been posed to higher education institution, particularly in Portugal. One of the main implications is related to the change of the paradigm of a teacher centered education, to a paradigm that is student centered. This change implies the change of the way to assess courses in higher education institutions. Continuous and formative assessments emerged as the focus, catalyzed by electronic assessment, or e-assessment. This paper presents a case of the implementation of an e-assessment strategy, implemented in order to allow continuous, formative assessment in numerous mathematics classes using multiple-choice questions tests implement in Moodle open-source learning management system. The implementation can be considered a success.

E-assessment in mathematics courses with multiple-choice questions tests

With the implementation of the Bologna Process several challenges have been posed to higher education institution, particularly in Portugal. One of the main implications is related to the change of the paradigm of a teacher centered education, to a paradigm that is student centered. This change implies the change of the way to assess courses in higher education institutions. Continuous and formative assessments emerged as the focus, catalyzed by electronic assessment, or e-assessment. This paper presents a case of the implementation of an e-assessment strategy, implemented in order to allow continuous, formative assessment in numerous mathematics classes using multiple-choice questions tests implement in Moodle open-source learning management system. The implementation can be considered a success.

Social justice in education : how the function of selection in educational institutions predicts support for (non)egalitarian assessment practices

Educational institutions are considered a keystone for the establishment of a meritocratic society. They supposedly serve two functions: an educational function that promotes learning for all, and a selection function that sorts individuals into different programs, and ultimately social positions, based on individual merit. We study how the function of selection relates to support for assessment practices known to harm vs. benefit lower status students, through the perceived justice principles underlying these practices. We study two assessment practices: normative assessment-focused on ranking and social comparison, known to hinder the success of lower status students-and formative assessment-focused on learning and improvement, known to benefit lower status students. Normative assessment is usually perceived as relying on an equity principle, with rewards being allocated based on merit and should thus appear as positively associated with the function of selection. Formative assessment is usually perceived as relying on corrective justice that aims to ensure equality of outcomes by considering students' needs, which makes it less suitable for the function of selection. A questionnaire measuring these constructs was administered to university students. Results showed that believing that education is intended to select the best students positively predicts support for normative assessment, through increased perception of its reliance on equity, and negatively predicts support for formative assessment, through reduced perception of its ability to establish corrective justice. This study suggests that the belief in the function of selection as inherent to educational institutions can contribute to the reproduction of social inequalities by preventing change from assessment practices known to disadvantage lowerstatus student, namely normative assessment, to more favorable practices, namely formative assessment, and by promoting matching beliefs in justice principles.

Assessment in CLIL

The aim of this talk is to explore assessment in higher education and support for learners during their academic education. It investigates the assessment methods that provide success for universities and learners. Universities which implement CLIL should assess learners who would like to attend a programme in English without taking account of their English language qualifications. This assessment should be done in writing, listening and comprehension, speaking and reading and comprehension. In the literature, formative and summative assessments are considered. Formative assessment is ongoing, 'more complex as its intention is to be directly diagnostics with a view to immediately impacting on learner's next steps' (Coyle et al, 2010). Summative assessment occurs at the end of the term or course. McKay, 2006 divides assessment into three phases: design, operationalization and administration phase. If these three phases can't be embedded in the classroom as they are, they can be introduced as a set of questions; why?, how? and what?

Façons de faire l’évaluation formative d’enseignants de français sénégalais : une analyse de leurs savoirs pratiques en contexte d’effectifs pléthoriques au Lycée.

Cette recherche documente les façons de faire l’évaluation formative d’enseignants de français du Lycée exerçant en contexte de classes pléthoriques au Sénégal. Le choix récent dans ce pays de l’approche par compétences invite à privilégier cette fonction de l’évaluation, au regard de son potentiel pour la progression des apprentissages des élèves (Allal & Mottier Lopez, 2005; Black & Wiliam, 2009; Morrissette, 2010). Cependant, les orientations ministérielles concernant sa mise en œuvre sont très générales, et jusqu’ici, la recherche a laissé dans l’ombre son application en contexte de classes pléthoriques. Puisant au domaine des savoirs pratiques (Schön, 1983) et à une vision interactive et située de l’évaluation formative (Mottier Lopez, 2007; Morrissette, 2010), j’ai conduit une démarche de recherche collaborative auprès de 14 enseignants de français exerçant dans le même Lycée, ponctuée par 6 entretiens de groupe. Un premier registre d’analyse a décrit des façons de faire rattachées à trois dimensions de la pratique de l’évaluation formative: l’analyse du contexte de la pratique, la construction négociée du savoir et la gestion de l’effectif. Un second registre d’analyse de leurs façons de faire en contexte d’«étrangeté culturelle» (Douville, 2002) a permis de conceptualiser leur savoir-évaluer en relation avec leur façon d’interpréter les problèmes qui se posent aux élèves, leur conception de l’erreur et leurs manières de réinventer les modes d’accomplissement traditionnels de l’évaluation ancrés dans la culture scolaire.

Development of a Real-time Formative Feedback Student Response System

This paper is focusing IT-supported real-time formative feedback in a classroom context. The development of a Student and Teacher Response System (STRS) is described. Since there are a number of obstacles for effective interaction in large classes, IT can be used to support the teachers aim to find out if students understand the lecture and accordingly adjust the content and design of the lecture. The system can be used for formative assessment before, during, and after a lecture. It is also possible for students to initiate interaction during lectures by posing questions anonymously. The main contributions of the paper are a) the description of the interactive real-time system and b) the development process behind it.

Effects of Self-Assessment on Math Homework

In this action research study of my eighth grade differentiated Algebra students, I investigated the effects of students using self-assessment on their homework. Students in my class were unmotivated and failed test objectives consistently. I wanted students to see that they controlled their learning and could be motivated to succeed. Formative assessment tells students how they need to improve. Learning needs to happen before they can be assessed. Self-assessment is one tool that helps students know if they are learning. A rubric scoring guide, daily documentation sheet and feedback on homework and test correlations were used to help students monitor their learning. Students needed time to develop the skill to self-assess. Students began to understand the relationship between homework and performing well on tests by the end of the action research period. Early in the period, most students encountered difficulty understanding that they controlled their learning and did not think homework was important. By the end of the year, all students said homework was important and that it helped them on quizzes and tests. Motivating students to complete homework is difficult. Teaching them to self-assess and to keep track of their learning helps them stay motivated.

An approach to automatic learning assessment based on the computational theory of perceptions

E-learning systems output a huge quantity of data on a learning process. However, it takes a lot of specialist human resources to manually process these data and generate an assessment report. Additionally, for formative assessment, the report should state the attainment level of the learning goals defined by the instructor. This paper describes the use of the granular linguistic model of a phenomenon (GLMP) to model the assessment of the learning process and implement the automated generation of an assessment report. GLMP is based on fuzzy logic and the computational theory of perceptions. This technique is useful for implementing complex assessment criteria using inference systems based on linguistic rules. Apart from the grade, the model also generates a detailed natural language progress report on the achieved proficiency level, based exclusively on the objective data gathered from correct and incorrect responses. This is illustrated by applying the model to the assessment of Dijkstra’s algorithm learning using a visual simulation-based graph algorithm learning environment, called GRAPHs

Peer Assessment and Self-assessment: Effective Learning Tools in Higher Education.

When used appropriately, self- and peer-assessment are very effective learning tools. In the present work, instructor formative assessment and feedback, self-assessment (SA), and peer-assessment (PA) have been compared. During the first part of a semester, the students followed a continuous formative assessment. Subsequently, they were divided into two subgroups based on similar performances. One subgroup performed SAs, and the other followedPAduring the last part of the course. The performances of the two groups in solving problems were compared. Results suggest that PA is a more effective learning tool than SA, and both are more effective than instructor formative assessment. However, a survey that was conducted at the end of the experiment showed higher student confidence in instructor assessment than in PA. The students recognized the usefulness of acting as peer assessors, but believed that SA helped them more than PA.