A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation.

OBJECTIVE: To pilot test if Orthopaedic Surgery residents could self-assess their performance using newly created milestones, as defined by the Accreditation Council on Graduate Medical Education. METHODS: In June 2012, an email was sent to Program Directors and administrative coordinators of the 154 accredited Orthopaedic Surgery Programs, asking them to send their residents a link to an online survey. The survey was adapted from the Orthopaedic Surgery Milestone Project. Completed surveys were aggregated in an anonymous, confidential database. SAS 9.3 was used to perform the analyses. RESULTS: Responses from 71 residents were analyzed. First and second year residents indicated through self-assessment that they had substantially achieved Level 1 and Level 2 milestones. Third year residents reported they had substantially achieved 30/41, and fourth year residents, all Level 3 milestones. Fifth year, graduating residents, reported they had substantially achieved 17 Level 4 milestones, and were extremely close on another 15. No milestone was rated at Level 5, the maximum possible. Earlier in training, Patient Care and Medical Knowledge milestones were rated lower than the milestones reflecting the other four competencies of Practice Based Learning and Improvement, Systems Based Practice, Professionalism, and Interpersonal Communication. The gap was closed by the fourth year. CONCLUSIONS: Residents were able to successfully self-assess using the 41 Orthopaedic Surgery milestones. Respondents' rate improved proficiency over time. Graduating residents report they have substantially, or close to substantially, achieved all Level 4 milestones. Milestone self-assessment may be a useful tool as one component of a program's overall performance assessment strategy.

How Good Am I? Crude Self-Assessment vs. Examination

With the new academic year structure encouraging more in-term assessment to replace end-of-year examinations one of the problems we face is assessing students and keeping track of individual student learning without overloading the students and staff with excessive assessment burdens.
In the School of Electronics, Electrical Engineering and Computer Science, we have constructed a system that allows students to self-assess their capability on a simple Yes/No/Don’t Know scale against fine grained learning outcomes for a module. As the term progresses students update their record as appropriately, including selecting a Learnt option to reflect improvements they have gained as part of their studies.
In the system each of the learning outcomes are linked to the relevant teaching session (lectures and labs) and to online resources that students can access at any time. Students can structure their own learning experience to their needs and preferences in order to attain the learning outcomes.
The system keeps a history of the student’s record, allowing the lecturer to observe how the students’ abilities progress over the term and to compare it to assessment results. The system also keeps of any of the resource links that student has clicked on and the related learning outcome.
The initial work is comparing the accuracy of the student self-assessments with their performance in the related questions in the traditional end-of-year examination.

COMP1205 self-assessment template CV

Appendix needed for the CV coursework.

Fetal Movements in late Pregnancy : Categorization, Self-assessment, and Prenatal Attachment in relation to women’s experiences

Aim: To explore how pregnant women experience fetal movements in late pregnancy. Specific aims were:  to study women’s experiences during the time prior to receiving news that their unborn baby had died in utero (I), to investigate women’s descriptions of fetal movements (II), investigate the association between the magnitude of fetal movements and level of prenatal attachment (III), and to study women’s experiences using two different self-assessment methods (IV). Methods: Interviews, questionnaires, and observations were used. Results: Premonition that something had happened to their unborn baby, based on a lack of fetal movements, was experienced by the participants. The overall theme “something is wrong” describes the women’s insight that the baby’s life was threatened (I). Fetal movements that were sorted into the domain “powerful movements” were perceived in late pregnancy by 96 % of the participants (II). Perceiving frequent fetal movements on at least three occasions per 24 hours was associated with higher scores of prenatal attachment in all the three subscales on PAI-R. The majority (55%) of the 456 participants reported average occasions of frequent fetal movements, 26% several occasions and 18% reported few occasions of frequent fetal movements, during the current gestational week.  (III). Only one of the 40 participants did not find at least one method for monitoring fetal movements suitable. Fifteen of the 39 participants reported a preference for the mindfetalness method and five for the count-to-ten method. The women described the observation of the movements as a safe and reassuring moment for communication with their unborn baby (IV). Conclusion:  In full-term and uncomplicated pregnancies, women usually perceive fetal movements as powerful. Furthermore, women in late pregnancy who reported frequent fetal movements on several occasions during a 24-hour period seem to have a high level of prenatal attachment. Women who used self-assessment methods for monitoring fetal movements felt calm and relaxed when observing the movements of their babies. They had a high compliance for both self-assessment methods. Women that had experienced a stillbirth in late pregnancy described that they had a premonition before they were told that their baby had died in utero. 

RESULTS OF THE SELF-ASSESSMENT AND DOCUMENTATION LIBRARY CAREER WITH A VIEW TO THE REACREDITATION, 2008

Succinctly defines the terms "accreditation" and "recertification" in order to locate the reader (a) on the theme to develop. Also describes the process of self-assessment of Librarianship and Documentation Race towards the re-accreditation, focusing primarily on results obtained in the development of self-evaluation reports of the various sectors that make up the School. Finally, presents some conclusions both about the overall process of the various reports as described.

Reaction Mechanisms Booster (Self-assessment resources)

This is a set of resources aimed at promoting the development of the skills required to successfully identify and generate organic mechanisms in order to demonstrate a good understanding of the underlying chemical principles. Students complete a task independently, and then mark their own work as they watch talking mark scheme videos where an expert explains how to get to the correct answers. Our research has shown that engagement with these resources is perceived to be highly beneficial by students, and leads to an increase in their confidence to tackle mechanistic problems. If you are a non-UK (or non-A-level) teacher, the terms AS and A2 may be meaningless to you, but the resources should still be useful for anyone studying organic reaction mechanisms. The worksheets are available in Word format, and you should feel free to edit these to meet the needs of your students and the course you are teaching. You can download the files individually by selecting them on the left and clicking download. Please watch the short briefing video on You Tube and read the teachers' notes carefully. Contact me on d.read@soton.ac.uk if you have and questions or comments. Additionally, I would like to thank Henry Pearson for suggesting the format of the alternative self-assessment proformas which you may choose to use with your students.

Investigating Formative Assessment: Exploring the Impact on the Self-Efficacy and Motivation of Mathematics Students

This is a study of the implementation and impact of formative assessment strategies on the motivation and self-efficacy of secondary school mathematics students. An explanatory sequential mixed methods design was implemented where quantitative and qualitative data were collected and analyzed sequentially in 2 different phases. The first phase involved quantitative data from student questionnaires and the second phase involved qualitative data from individual student and teacher interviews. The findings of the study suggest that formative assessment is implemented in practice in diverse ways and is a process where the strategies are interconnected. Teachers experience difficulty in incorporating peer and self-assessment and perceive a need for exemplars. Key factors described as influencing implementation include teaching philosophies, interpretation of ministry documents, teachers’ experiences, leadership in administration and department, teacher collaboration, misconceptions of teachers, and student understanding of formative assessment. Findings suggest that overall, formative assessment positively impacts student motivation and self-efficacy, because feedback is provided which offers encouragement and recognition by highlighting the progress that has been made and what steps need to be taken to improve. However, students are impacted differently with some considerations including how students perceive mistakes and if they fear judgement. Additionally, the impact of formative assessment is influenced by the connection between self-efficacy and motivation, namely how well a student is doing is a source of both concepts.

Health-related quality of life in Huntington’s Disease patients: a comparison of proxy assessment and patient self-rating using the disease-specific Huntington’s Disease health-related quality of life questionnaire (HDQoL)

Huntington’s disease (HD) is a fatal, neurodegenerative disease for which there is no known cure. Proxy evaluation is relevant for HD as its manifestation might limit the ability of persons to report their health-related quality of life (HrQoL). This study explored patient–proxy ratings of HrQoL of persons at different stages of HD, and examined factors that may affect proxy ratings. A total of 105 patient–proxy pairs completed the Huntington’s disease health-related quality of life questionnaire (HDQoL) and other established HrQoL measures (EQ-5D and SF-12v2). Proxy–patient agreement was assessed in terms of absolute level (mean ratings) and intraclass correlation. Proxies’ ratings were at a similar level to patients’ self-ratings on an overall Summary Score and on most of the six Specific Scales of the HDQoL. On the Specific Hopes and Worries Scale, proxies on average rated HrQoL as better than patients’ self-ratings, while on both the Specific Cognitive Scale and Specific Physical and Functional Scale proxies tended to rate HrQoL more poorly than patients themselves. The patient’s disease stage and mental wellbeing (SF-12 Mental Component scale) were the two factors that primarily affected proxy assessment. Proxy scores were strongly correlated with patients’ self-ratings of HrQoL, on the Summary Scale and all Specific Scales. The patient–proxy correlation was lower for patients at moderate stages of HD compared to patients at early and advanced stages. The proxy report version of the HDQoL is a useful complementary tool to self-assessment, and a promising alternative when individual patients with advanced HD are unable to self-report.

Elements of Better Assessment for the Improvement of Learning

Building a community of shared practice at the classroom level calls for clarity about the important assessment capabilities and dispositions of teachers, especially when teachers are expected to take a direct focus on learning. In this chapter, we present new ways of thinking about teachers’ assessment literacies, offering a formulation of better assessment for the improvement of learning, including three elements, namely (i) assessment criteria and standards; (ii) the teacher’s professional judgment; and (iii) social moderation. The potential of the first element lies in teachers’ classroom practices that deliberately embed assessment criteria and standards in pedagogy in productive ways. The second element involves the engagement of teachers and students in judgment practice, that develops the understanding that judgment involves more than the application of explicit or stated criteria. More fundamental is the matter of how teachers bring to bear stated features of quality and other intellectual and experiential resources in arriving at judgment. That is to say, they range across and orient to explicit (stated), tacit (unstated) and meta-criteria in judgment making. These insights have direct relevance to teachers’ efforts to develop students’ own evaluative experience, especially as this involves students working with stated features of quality for self-assessment and peer-assessment purposes. Further, practices for social moderation are discussed, giving examples of good practice in moderation, how teachers experience moderation and the potential benefits of various types.

Computer usage and the validity of self-assessed computer competence among first-year business students

Abstract This study evaluates the reliability of self-assessment as a measure of computer competence. This evaluation is carried out in response to recent research which has employed self-reported ratings as the sole indicator of students’ computer competence. To evaluate the reliability of self-assessed computer competence, the scores achieved by students in self-assessed computer competence tests are compared with scores achieved in objective tests. The results reveal a statistically significantly over-estimation of computer competence among the students surveyed. Furthermore, reported pre-university computer experience in terms of home and school use and formal IT education does not affect this result. The findings call into question the validity of using self-assessment as a measure of computer competence. More generally, the study also provides an up-to-date picture of self-reported computer usage and IT experience among pre-university students from New Zealand and South-east Asia and contrasts these findings with those from previous research.

Assessment of fighting ability in animal contests

Selection should favour accurate information gathering regarding the likely costs and benefits of continued conflict. Here we consider how variation in the abilities of contestants to assess resource-holding potential (RHP) influences fights. This has been examined in various game theory models. However, discriminating between assessment strategies has proven difficult and has resulted in confusion. To add clarity, we group existing models into three main types that differ in the information about RHP that contestants are presumed to gather: (1) pure self-assessment, (2) cumulative assessment and (3) mutual assessment. Within this framework we outline methods advocated to discriminate successfully between the three main assessment models. We discuss support for each model, before highlighting a number of conflicting and inconclusive studies, leading us to consider alternative approaches to investigate assessment. Furthermore, we examine support for newly emerging concepts such as 'varying degrees of assessment', 'switching assessment' strategies and the possibility of contestants adopting different assessment strategies within a fight involving distinctive roles. We suggest future studies will benefit by judicious use of a battery of techniques to determine how animals settle contests. Finally, we highlight difficulties with current game theory models, and raise concerns regarding the use of certain behavioural criteria to accept or reject a model, particularly since this may conflict with evidence for a given assessment strategy. Furthermore, the failure of existing models to account for newly emerging concepts points to limitations of their use and leads us to challenge game theoreticians to develop upon them. (C) 2009 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.

Vocal rate as an assessment process during fallow deer contests

Two types of model propose that strategic decisions during contests are determined either by (i) a mutual-assessment process or (ii) a self-assessment process. Vocal signals are thought to convey information about the competitive abilities of individuals, the ultimate function of which is a reduction in costs associated with fighting consistent with the principle of mutual assessment. Nevertheless, the limited evidence that male ungulates engage in mutual assessment of vocal rates during dyadic contests has been questioned. Therefore, we examined the vocal rates of winners and losers during escalated dyadic contests between male fallow deer in order to further inform on this issue. Our results showed that winners and losers did not differ in vocal rate. The best model fit that accounted for individual vocal rates included a preponderance of factors related to the opponent indicating that contestants were attending to their opponent during fights. Vocal rate was, therefore, dependent on estimates of opponent quality without reference to self, supporting an 'opponent-only' rather than a mutual assessment process. © 2012 Elsevier B.V.

Abstract: Psychometric properties of two self-report clinical competence scales for nursing students.

Abstract: Psychometric properties of two self-report clinical competence scales for nursing students.
Background: It is important to assess the clinical competence of nursing students to gauge their professional development and educational needs. This can be measured by self-assessment tools. Anema and McCoy (2010) contended that the currently available measures need further psychometric testing.
Aim: To test the psychometric properties of Nursing Competencies Questionnaire (NCQ) and Self-Efficacy in Clinical Performance (SECP) clinical competence scales.

Method: A non-randomly selected sample of n=248 2nd year nursing students completed NCQ, SECP and demographic questionnaires (June and September 2013). Mokken Scaling Analysis (MSA) was used to test the structural validity and scale properties, convergent and discriminant validity and reliability were subsequently tested.

Results: The NCQ provided evidence of a unidimensional scale which had strong scale scalability coefficients Hs =0.581; but limited evidence of item rankability HT =0.367. MSA undertaken with the SECP scale identified two potential unidimensional scales the SECP28 and SECP7, each with adequate evidence of good/reasonable scalablity psychometric properties as a summed scale but no/very limited evidence of scale rankability (SECP28: Hs = 0.55, HT=0.211; SECP7: Hs = 0.61, HT=0.049). Analysis of between cohort differences and NCQ/ SECP scale scores produced evidence of convergent and discriminant validity and good internal reliability: NCQ α = 0.93, SECP28 α = 0.96, and SECP7 α=0.89.

Discussion: The NCQ was verified to have evidence of reliability and validity; however, as the SECP findings are new, and the sample small, with reference to Straat and colleagues (2014), the SECP results should be interpreted with caution and verified on a second sample.

Conclusions: Measurement of perceived self-competence could inform the development of nursing competence and could start early in a nursing programme. Further testing of the NCQ and SECP scales with larger samples and from different years is indicated.


References:
Anema, M., G and McCoy, JK. (2010) Competency-Based Nursing Education: Guide to Achieving Outstanding Learner Outcomes. New York: Springer.
Straat, JH., van der Ark, LA and Sijtsma, K. (2014) Minimum Sample Size Requirements for Mokken Scale Analysis Educational and Psychological Measurement 74 (5), 809-822.

A Qualidade na administração pública e a CAF – Common assessment framework

Apresentação no âmbito da Dissertação de Mestrado Orientador: Doutora Alcina Dias

A Teacher’s Formative Assessment Perceptions and Practices in Oral Intermediate English Courses at the Université de Montréal

Au cours des dernières années, les pratiques de l’évaluation comme aide à l’apprentissage (c’est-à-dire l’auto-évaluation, l’évaluation par les pairs, la rétroaction) dans la salle de classe ont été de plus en plus considérées comme des éléments essentiels (Black & Wiliam, 1998; Harlen & Winter, 2004; van de Watering & van der Rijt, 2006). Cependant, dans le domaine de l’apprentissage d’une langue seconde la recherche sur ce sujet est plutôt limitée. En nous fondant sur les études de Colby-Kelly et Turner (2007) et de Lyster et Ranta (1997), nous avons mené une recherche exploratoire visant à combler ce besoin. L’objectif général était de comprendre comment l’évaluation formative se réalise dans deux cours d’anglais intermédiaire à l’Université de Montréal, et de comparer comment ces pratiques sont perçues et vécues par la professeure et ses étudiants. Trois questions de recherche étaient posées: 1. Quelle est la nature de l’évaluation formative dans une salle de classe d’anglais langue seconde? 2. Est-ce que les pratiques de la professeure reflètent ce qu’elle pense de l’évaluation formative? 3. Quels sont les correspondances et différences entre les croyances de la professeure et les perceptions de ses étudiants quant aux bénéfices de l’évaluation formative pour l’apprentissage de l’anglais langue seconde? La collecte de données comprend des observations en classe, des entrevues auprès de la professeure et des questionnaires pour les étudiants. Pour répondre à notre première question de recherche, nous avons analysé la rétroaction donnée par la professeure pendant une production orale réalisée par ses étudiants à partir de la typologie de feedback et de uptake de l’étude de Lyster et Ranta (1997). En ce qui a trait à la deuxième question de recherche, nous avons fait des entrevues avec la professeure en vue d’expliciter ses perceptions relativement à l’évaluation formative. À la fin du trimestre, nous avons comparé ses réponses avec sa pratique à partir des enregistrements vidéo produits en classe. Finalement, pour répondre à notre dernière question de recherche nous avons comparé les réponses données par la professeure aux entrevues avec les réponses des étudiants à un questionnaire adapté à partir de celui de Colby-Kelly et Turner. Finalement, nous présentons et discutons les résultats les plus significatifs obtenus dans cette étude qualitative Nous concluons cette thèse en proposant de avenues pour des recherches futures.