The use of scoring rubrics to determine clinical performance in the operating suite

Summary : This research evolved out of the need to examine the validity and inter-rater reliability of a set of performance-based scoring rubrics designed to measure competencies within the operating suite.

Method : Both holistic and analytical rubrics were developed aligned to the ACORN Standard [Australian College of Operating Room Nurses Standard NR4, 2004. ACORN Competency Standards for Perioperative Nurses: Standard NR4: The Instrument Nurse in the Perioperative Environment. Australian College of Operating Room Nurses Ltd, Adelaide] and underpinned by the Dreyfus model (1981). Three video clips that captured varying performance of nurses performing as instrument nurses in the operating suite were recorded and used as prompts by expert raters, who judged the performance using the rubrics.

Results : The study found that the holistic rubrics led to more consistent judgments than the analytical rubrics, yet the latter provided more diagnostic information for intervention purposes. Despite less consistency, the Analytical Observation Form had sufficient construct validity to satisfy the requirements of criterion referencing as determined by the Item Separation Index (Rasch, 1960), including high internal consistency and greater inter-rater reliability when average ratings were used.

Conclusion : The study was an empirical investigation of the use of concomitant Analytical and Holistic Rubrics to determine various levels of performance in the operating suite including inter-rater reliability. The methodology chosen was theoretically sound and sufficiently flexible to be used to develop other competencies within the operating suite.

Authentic and conventional assessment in Singapore schools : an empirical study of teacher assignments and student work

This is an empirical examination of the quality of teacher assignments and student work in Singapore schools. Using a theoretical framework based on principles of authentic assessment and intellectual quality, two sets of criteria and scoring rubrics were developed for the training of expert teachers to judge the quality of assignments and student work. Following rigorous training, the inter-rater reliability of expert teacher scoring was high. Samples of teacher assignments and student work were collected in English, social studies, mathematics, and science subject areas from a random stratified sample of 30 elementary schools and 29 high schools. For both grade levels, there were significant differences for the authentic intellectual quality of teachers’ assignments by subject area. Likewise, the differences of authentic intellectual quality for student work were significant and varied by subject area. Subject area effect was large. The correlations between the quality of teachers’ assignment tasks and student work were strong and significant at both grade levels. Where teachers set more intellectually demanding tasks, students were more likely to generate work or artefacts judged to be of higher quality. The findings suggest that teacher professional development in authentic intellectual assessment task design can contribute to the improvement of student learning and performance. It is argued that this will be a key requisite of educational systems like Singapore that are seeking to expand pedagogy and student outcomes beyond a focus on factual and rote knowledge.

La carpeta de aprendizaje y la evaluación entre iguales de los estudiantes. Resultados de una innovación docente en los estudios universitarios de las artes

Students reflect more on their learning in course subjects when they participate in managing their teaching–learning environment. As a form of guided participation, peer assessment serves the following purposes: (a) it improves the student’s understanding of previously established learning objectives; (b) it is a powerful metacognitive tool; (c) it transfers to the student part of the responsibility for assessing learning, which means deciding which learning activities are important and choosing the degree of effort a course subject will require; (d) it emphasizes the collective aspect of the nature of knowledge; and (e) the educational benefits derived from peer assessment clearly justify the efforts required to implement activities. This paper reports on the relative merits of a learning portfolio compiled during fine arts-related studies in which peer assessment played an important role. The researchers analyzed the student work load and the final marks students received for compulsory art subjects. They conclude that the use of a closed learning portfolio with a well-structured, sequential and analytical design can have a positive effect on student learning and that, although implementing peer assessment may be complex and students need to become familiar with it, its use is not only feasible but recommendable.

La importancia de la formación del profesorado en la evaluación de aprendizajes con rúbrica

En esta investigación pretendemos conocer si las emociones que experimenta el alumnado de Primaria al ser evaluado con rúbrica pueden verse afectadas por la formación y el conocimiento que sobre dicho instrumento se les presupone a sus evaluadores. Por ello, se establece una comparativa entre las emociones que sienten los estudiantes al ser evaluados por profesores (a los que se les reconoce una formación sobre rúbrica) y las que sienten aquellos que son coevaluados por sus iguales o autoevaluados. Un total de 66 alumnos del 2º ciclo de Enseñanza Básica (10 años y 11 años) de Évora (Portugal), que habían tenido experiencia de heteroevaluación, autoevaluación o coevaluación con rúbrica, cumplimentaron un amplio cuestionario. Los resultados muestran que cerca de un 62% de los estudiantes que fueron evaluados por pares o autoevaluados a través de una rúbrica experimentan emociones positivas, frente a un 91 % de aquellos cuyo trabajo fue valorado por profesores utilizando dicho instrumento. En consonancia con este dato, existe mayor probabilidad de experimentar emociones positivas cuando se recibe una evaluación con rúbrica por parte de un maestro que cuando la valoración procede de un igual o de uno mismo. También se ha observado que el porcentaje de estudiantes que experimenta emociones negativas es mayor cuando no son evaluados por el profesor. Además, un 76 % de los alumnos evaluados con rúbrica por los compañeros considera que esta acción puede ayudarles, frente a un 81% de los evaluados por el docente que comparten esta opinión. Por último, cabe destacar la posibilidad de que estos resultados estén mediatizados por la percepción de que una evaluación realizada por un igual no posee la misma calidad que la que lleva a cabo un profesor.

Assessment rubrics: towards clearer and more replicable design, research and practice

‘Rubric’ is a term with a variety of meanings. As the use of rubrics has increased both in research and practice, the term has come to represent divergent practices. These range from secret scoring sheets held by teachers to holistic student-developed articulations of quality. Rubrics are evaluated, mandated, embraced and resisted based on often imprecise and inconsistent understandings of the term. This paper provides a synthesis of the diversity of rubrics, and a framework for researchers and practitioners to be clearer about what they mean when they say ‘rubric’. Fourteen design elements or decision points are identified that make one rubric different from another. This framework subsumes previous attempts to categorise rubrics, and should provide more precision to rubric discussions and debate, as well as supporting more replicable research and practice.

Comparison of scoring systems for invasive pests using ROC analysis and Monte Carlo simulation

Different international plant protection organisations advocate different schemes for conducting pest risk assessments. Most of these schemes use structured questionnaire in which experts are asked to score several items using an ordinal scale. The scores are then combined using a range of procedures, such as simple arithmetic mean, weighted averages, multiplication of scores, and cumulative sums. The most useful schemes will correctly identify harmful pests and identify ones that are not. As the quality of a pest risk assessment can depend on the characteristics of the scoring system used by the risk assessors (i.e., on the number of points of the scale and on the method used for combining the component scores), it is important to assess and compare the performance of different scoring systems. In this article, we proposed a new method for assessing scoring systems. Its principle is to simulate virtual data using a stochastic model and, then, to estimate sensitivity and specificity values from these data for different scoring systems. The interest of our approach was illustrated in a case study where several scoring systems were compared. Data for this analysis were generated using a probabilistic model describing the pest introduction process. The generated data were then used to simulate the outcome of scoring systems and to assess the accuracy of the decisions about positive and negative introduction. The results showed that ordinal scales with at most 5 or 6 points were sufficient and that the multiplication-based scoring systems performed better than their sum-based counterparts. The proposed method could be used in the future to assess a great diversity of scoring systems.

The effect of implementing a modified early warning scoring (MEWS) system on the adequacy of vital sign documentation

Introduction and objectives Early recognition of deteriorating patients results in better patient outcomes. Modified early warning scores (MEWS) attempt to identify deteriorating patients early so timely interventions can occur thus reducing serious adverse events. We compared frequencies of vital sign recording 24 h post-ICU discharge and 24 h preceding unplanned ICU admission before and after a new observation chart using MEWS and an associated educational programme was implemented into an Australian Tertiary referral hospital in Brisbane. Design Prospective before-and-after intervention study, using a convenience sample of ICU patients who have been discharged to the hospital wards, and in patients with an unplanned ICU admission, during November 2009 (before implementation; n = 69) and February 2010 (after implementation; n = 70). Main outcome measures Any change in a full set or individual vital sign frequency before-and-after the new MEWS observation chart and associated education programme was implemented. A full set of vital signs included Blood pressure (BP), heart rate (HR), temperature (T°), oxygen saturation (SaO2) respiratory rate (RR) and urine output (UO). Results After the MEWS observation chart implementation, we identified a statistically significant increase (210%) in overall frequency of full vital sign set documentation during the first 24 h post-ICU discharge (95% CI 148, 288%, p value <0.001). Frequency of all individual vital sign recordings increased after the MEWS observation chart was implemented. In particular, T° recordings increased by 26% (95% CI 8, 46%, p value = 0.003). An increased frequency of full vital sign set recordings for unplanned ICU admissions were found (44%, 95% CI 2, 102%, p value = 0.035). The only statistically significant improvement in individual vital sign recordings was urine output, demonstrating a 27% increase (95% CI 3, 57%, p value = 0.029). Conclusions The implementation of a new MEWS observation chart plus a supporting educational programme was associated with statistically significant increases in frequency of combined and individual vital sign set recordings during the first 24 h post-ICU discharge. There were no significant changes to frequency of individual vital sign recordings in unplanned admissions to ICU after the MEWS observation chart was implemented, except for urine output. Overall increases in the frequency of full vital sign sets were seen.

Scoring-thresholding pattern based text classifier

A big challenge for classification on text is the noisy of text data. It makes classification quality low. Many classification process can be divided into two sequential steps scoring and threshold setting (thresholding). Therefore to deal with noisy data problem, it is important to describe positive feature effectively scoring and to set a suitable threshold. Most existing text classifiers do not concentrate on these two jobs. In this paper, we propose a novel text classifier with pattern-based scoring that describe positive feature effectively, followed by threshold setting. The thresholding is based on score of training set, make it is simple to implement in other scoring methods. Experiment shows that our pattern-based classifier is promising.

Prognostic factors for malignant mesothelioma in 142 patients : validation of CALGB and EORTC prognostic scoring systems

Background The incidence of malignant mesothelioma is increasing. There is the perception that survival is worse in the UK than in other countries. However, it is important to compare survival in different series based on accurate prognostic data. The European Organisation for Research and Treatment of Cancer (EORTC) and the Cancer and Leukaemia Group B (CALGB) have recently published prognostic scoring systems. We have assessed the prognostic variables, validated the EORTC and CALGB prognostic groups, and evaluated survival in a series of 142 patients. Methods Case notes of 142 consecutive patients presenting in Leicester since 1988 were reviewed. Univariate analysis of prognostic variables was performed using a Cox proportional hazards regression model. Statistically significant variables were analysed further in a forward, stepwise multivariate model. EORTC and CALGB prognostic groups were derived, Kaplan-Meier survival curves plotted, and survival rates were calculated from life tables. Results Significant poor prognostic factors in univariate analysis included male sex, older age, weight loss, chest pain, poor performance status, low haemoglobin, leukocytosis, thrombocytosis, and non-epithelial cell type (p<0.05). The prognostic significance of cell type, haemoglobin, white cell count, performance status, and sex were retained in the multivariate model. Overall median survival was 5.9 (range 0-34.3) months. One and two year survival rates were 21.3% (95% CI 13.9 to 28.7) and 3.5% (0 to 8.5), respectively. Median, one, and two year survival data within prognostic groups in Leicester were equivalent to the EORTC and CALGB series. Survival curves were successfully stratified by the prognostic groups. Conclusions This study validates the EORTC and CALGB prognostic scoring systems which should be used both in the assessment of survival data of series in different countries and in the stratification of patients into randomised clinical studies.

Scoring rules and abnormally low bids criteria in construction tenders: A taxonomic review

In the global construction context, the Best Value or Most Economically Advantageous Tender is becoming a widespread approach for contractor selection, as an alternative to other traditional awarding criteria such as the Lowest Price. In these multi-attribute tenders, the owner or auctioneer solicits proposals containing both a price bid and additional technical features. Once the proposals are received, each bidder's price bid is given an economic score according to a scoring rule, generally called an Economic Scoring Formula (ESF) and a technical score according to pre-specified criteria. Eventually, the contract is awarded to the bidder with the highest weighted overall score (economic + technical). However, Economic Scoring Formula selection by auctioneers is invariably and paradoxically a highly intuitive process in practice, involving few theoretical or empirical considerations, despite having being considered traditionally and mistakenly as objective, due to its mathematical nature. This paper provides a taxonomic classification of a wide variety of ESF and Abnormally Low Bid Criteria (ALBC) gathered in several countries with different tendering approaches. Practical implications concern the optimal design of price scoring rules in construction contract tenders, as well as future analyses of the effects of ESF and ALBC on competitive bidding behaviour.

Interpersonal coordination tendencies supporting the creation/prevention of goal scoring opportunities in futsal

Research on 1vs1 sub-phases in team sports has shown how one player coordinates his/her actions with his/her opponent and the location of a target/goal to attain performance objectives. In this study, we extended this approach to analysis of 5vs5 competitive performance in the team sport of futsal to provide a performance analysis framework that explains how players coordinate their actions to create/prevent opportunities to score goals. For this purpose, we recorded all 10 futsal matches of the 2009 Lusophony Games held in Lisbon. We analysed the displacement trajectories of a shooting attacker and marking defender in plays ending in a goal, a goalkeeper's save, and a defender's interception, at four specific moments during performance: (1) assisting attacker's ball reception; (2) moment of passing; (3) shooter's ball reception, and; (4) shot on goal. Statistical analysis showed that when a goal was scored, the defender's angle to the goal and to the attacker tended to decrease, the attacker was able to move to the same distance to the goal alongside the defender, and the attacker was closer to the defender and moving at the same velocity (at least) as the defender. This study identified emergent patterns of coordination between attackers and defenders under key competitive task constraints, such as the location of the goal, which supported successful performance in futsal.