Strategies to facilitate grading consistency in large classes

First year undergraduate university classes can be very large, and big student numbers often creates a challenge for instructors to ensure assignments are graded consistently across the cohort. This session describes and demonstrates the use of interactive audience response technology (ART) with assessors (rather than students) to moderate assignment grading. Results from preliminary research indicate this method of moderating the grading of assignments is effective, and achieves more consistent outcomes for students.

Using technology to facilitate grading consistency in large classes

University classes in marketing are often large, and therefore require teams of teachers to cover all of the necessary activities. A major problem with teaching teams is the inconsistency that results from myriad individuals offering subjective opinions. This innovation uses the latest moderation techniques along with Audience Response Technology (ART) to enhance the learning experience by providing more consistent and reliable grading in large classes. Assessment items are moderated before they are graded in meetings that employ ART. Results show the process is effective when the teaching team is very large, or there is a diverse range of experienced and inexperienced teachers. This “behind the scenes” innovation is not immediately apparent to students, but results in more consistent grades, more useful feedback for students, and more confident graders.

Accuracy and consistency of the visual diagnosis of exposed dentine on worn occlusal/incisal surfaces

Most indices for the assessment of wear of various aetiologies include the distinction between 'enamel still present' and 'dentine exposed' for grading. Since the visual diagnosis of exposed dentine has not yet been validated, the present study is a first attempt to investigate its accuracy and consistency. Sixty-one examiners (23 scientists, 18 university dentists and 20 dental students) were asked to diagnose 49 tooth areas with different grades of wear and to decide whether dentine was exposed (positive test) or not (negative test). Afterwards, the teeth were histologically evaluated. In 44 areas, dentine (also in all cases with minor wear) was exposed, and in 5 areas enamel was present. Overall sensitivity was 0.65, specificity 0.88 and the proportion of correct diagnoses was 0.67. The diagnosis 'dentine is exposed' was about 5 times as likely and the diagnosis 'dentine is not exposed' half as likely to come from an area with exposed dentine than from an enamel-covered area. The closeness of the visual diagnosis to the histological findings was only fair (kappa=0.27), no significant impact of professional experience was found. For inter- and intra-examiner agreement, kappa was 0.28 and 0.55, respectively. It was concluded that the diagnosis of exposed dentine is difficult.

A new method to facilitate valid and consistent grading cardiac events in childhood cancer survivors using medical records

BACKGROUND Cardiac events (CEs) are among the most serious late effects following childhood cancer treatment. To establish accurate risk estimates for the occurrence of CEs it is essential that they are graded in a valid and consistent manner, especially for international studies. We therefore developed a data-extraction form and a set of flowcharts to grade CEs and tested the validity and consistency of this approach in a series of patients. METHODS The Common Terminology Criteria for Adverse Events version 3.0 and 4.0 were used to define the CEs. Forty patients were randomly selected from a cohort of 72 subjects with known CEs that had been graded by a physician for an earlier study. To establish whether the new method was valid for appropriate grading, a non-physician graded the CEs by using the new method. To evaluate consistency of the grading, the same charts were graded again by two other non-physicians, one with receiving brief introduction and one with receiving extensive training on the new method. We calculated weighted Kappa statistics to quantify inter-observer agreement. RESULTS The inter-observer agreement was 0.92 (95% CI 0.80-1.00) for validity, and 0.88 (0.79-0.98) and 0.99 (0.96-1.00) for consistency with the outcome assessors who had the brief introduction and the extensive training, respectively. CONCLUSIONS The newly developed standardized method to grade CEs using data from medical records has shown excellent validity and consistency. The study showed that the method can be correctly applied by researchers without a medical background, provided that they receive adequate training.

Property cash flow studies : focusing on model consistency and data accuracy

The high degree of variability and inconsistency in cash flow study usage by property professionals demands improvement in knowledge and processes. Until recently limited research was being undertaken on the use of cash flow studies in property valuations but the growing acceptance of this approach for major investment valuations has resulted in renewed interest in this topic. Studies on valuation variations identify data accuracy, model consistency and bias as major concerns. In cash flow studies there are practical problems with the input data and the consistency of the models. This study will refer to the recent literature and identify the major factors in model inconsistency and data selection. A detailed case study will be used to examine the effects of changes in structure and inputs. The key variable inputs will be identified and proposals developed to improve the selection process for these key variables. The variables will be selected with the aid of sensitivity studies and alternative ways of quantifying the key variables explained. The paper recommends, with reservations, the use of probability profiles of the variables and the incorporation of this data in simulation exercises. The use of Monte Carlo simulation is demonstrated and the factors influencing the structure of the probability distributions of the key variables are outline. This study relates to ongoing research into functional performance of commercial property within an Australian Cooperative Research Centre.

Improving behaviour classification consistency: a technique from biological taxonomy

Quantitative behaviour analysis requires the classification of behaviour to produce the basic data. In practice, much of this work will be performed by multiple observers, and maximising inter-observer consistency is of particular importance. Another discipline where consistency in classification is vital is biological taxonomy. A classification tool of great utility, the binary key, is designed to simplify the classification decision process and ensure consistent identification of proper categories. We show how this same decision-making tool - the binary key - can be used to promote consistency in the classification of behaviour. The construction of a binary key also ensures that the categories in which behaviour is classified are complete and non-overlapping. We discuss the general principles of design of binary keys, and illustrate their construction and use with a practical example from education research.

Dancing between diversity and consistency : refining assessment in postgraduate degrees in dance

This publication is the culmination of a 2 year Australian Learning and Teaching Council's Project Priority Programs Research Grant which investigates key issues and challenges in developing flexible guidelines lines for best practice in Australian Doctoral and Masters by Research Examination, encompassing the two modes of investigation, written and multi-modal (practice-led/based) theses, their distinctiveness and their potential interplay. The aims of the project were to address issues of assessment legitimacy raised by the entry of practice-orientated dance studies into Australian higher degrees; examine literal embodiment and presence, as opposed to cultural studies about states of embodiment; foreground the validity of questions around subjectivity and corporeal intelligence/s and the reliability of artistic/aesthetic communications, and finally to celebrate ‘performance mastery’(Melrose 2003) as a rigorous and legitimate mode of higher research. The project began with questions which centred around: the functions of higher degree dance research; concepts of 'master-ness’ and ‘doctorateness’; the kinds of languages, structures and processes which may guide candidates, supervisors, examiners and research personnel; the purpose of evaluation/examination; addressing positive and negative attributes of examination. Finally the study examined ways in which academic/professional, writing/dancing, tradition/creation and diversity/consistency relationships might be fostered to embrace change. Over two years, the authors undertook a qualitative national study encompassing a triangulation of semi-structured face to face interviews and industry forums to gather views from the profession, together with an analysis of existing guidelines, and recent literature in the field. The most significant primary data emerged from 74 qualitative interviews with supervisors, examiners, research deans and administrators, and candidates in dance and more broadly across the creative arts. Qualitative data gathered from the two primary sources, was coded and analysed using the NVivo software program. Further perspectives were drawn from international consultant and dance researcher Susan Melrose, as well as publications in the field, and initial feedback from a draft document circulated at the World Dance Alliance Global Summit in July 2008 in Brisbane. Refinement of data occurred in a continual sifting process until the final publication was produced. This process resulted in a set of guidelines in the form of a complex dynamic system for both product and process oriented outcomes of multi-modal theses, along with short position papers on issues which arose from the research such as contested definitions, embodiment and ephemerality, ‘liveness’ in performance research higher degrees, dissolving theory/practice binaries, the relationship between academe and industry, documenting practices and a re-consideration of the viva voce.

A Review of Customer Involvement in Evaluations of Case Management : Consistency with a recovery paradigm

This Open Forum examines research on case management that draws on consumer perspectives. It clarifies the extent of consumer involvement and whether evaluations were informed by recovery perspectives. Searches of three databases revealed l3 studies that sought to investigate consumer perspectives. Only one study asked consumers about experiences of recovery. Most evaluations did not adequately assess consumers' views, and active consumer participation in research was rare. Supporting an individual's recovery requires commitment to a recovery paradigm that incorporates traditional symptom reduction and improved functioning, with broader recovery principles, and a shift in focus from illness to wellbeing. It also requires greater involvement of consumers in the implementation of case management and ownership of their own recovery process, not just in research that evaluates the practice.

Measuring Visual Consistency in 3D Rendering Systems

One of the major challenges facing a present day game development company is the removal of bugs from such complex virtual environments. This work presents an approach for measuring the correctness of synthetic scenes generated by a rendering system of a 3D application, such as a computer game. Our approach builds a database of labelled point clouds representing the spatiotemporal colour distribution for the objects present in a sequence of bug-free frames. This is done by converting the position that the pixels take over time into the 3D equivalent points with associated colours. Once the space of labelled points is built, each new image produced from the same game by any rendering system can be analysed by measuring its visual inconsistency in terms of distance from the database. Objects within the scene can be relocated (manually or by the application engine); yet the algorithm is able to perform the image analysis in terms of the 3D structure and colour distribution of samples on the surface of the object. We applied our framework to the publicly available game RacingGame developed for Microsoft(R) Xna(R). Preliminary results show how this approach can be used to detect a variety of visual artifacts generated by the rendering system in a professional quality game engine.

Developing consistency in teacher judgment formation through online meeting centres

This paper investigates the complex interactions that occur as teachers meet online to justify and negotiate their assessment judgments of student work across relatively large and geographically dispersed populations. Drawing from sociocultural theories of learning and technology, the technology is positioned as playing a role in either supporting or hindering teachers reaching a common understanding of assessment standards. Meeting transcripts and interviews with the teachers have been qualitatively analysed in terms of the interactions that occurred and teachers’ perceptions of these interactions. While online meetings offer a partial solution to address the current demands of assessment in education, they also present new challenges as teachers meet, in an unfamiliar environment, to discuss student work.

Automatic grading and packing of prawns

Prawns are a substantial Australian resource but presently are processed in a very labour-intensive manner. A prototype system has been developed for automatically grading and packing prawns into single-layer 'consumer packs' in which each prawn is approximately straight and has the same orientation. The novel technology includes a machine vision system that has been specially programmed to calculate relevant parameters at high speed and a gripper mechanism that can acquire, straighten and place prawns of various sizes. The system can be implemented on board a trawler or in an onshore processing facility. © 1993.

A survey of the use of grading scales for contact lens complications in optometric practice

Purpose:  The aim of this study was to investigate the extent and pattern of use of grading scales for contact lens complications (‘grading scales’) in optometric practice. Methods:  An anonymous postal survey was sent to all 756 members of the Queensland Division of Optometrists Association Australia. Information was elicited relating to level of experience, practice type and location, and mode of usage of grading scales. Results:  Survey forms were returned by 237 optometrists, representing a 31 per cent response rate. The majority of respondents (61 per cent) reported using grading scales frequently in practice, while 65 per cent of these preferred to use the Efron Grading Scales for Contact Lens Complications. Seventy-six per cent of optometrists use a method of incremental grading rather than simply grading with whole numbers. Grading scales are more likely to be used by optometrists who have recently graduated (p < 0.001), have a postgraduate certificate in ocular therapeutics (p = 0.018), see more contact lens patients (p = 0.027) and use other forms of grading scales (p < 0.001). The most frequently graded ocular conditions were corneal staining, papillary conjunctivitis and conjunctival redness. The main reasons for not using grading scales included a preference for sketches, photographs or descriptions (87 per cent) and unavailability of scales (29 per cent). Conclusion:  Grading scales for contact lens complications are used extensively in optometric practice for a variety of purposes. This tool can now be considered as an expected norm in contact lens practice. We advocate the incorporation of such grading scales into professional guidelines and standards for good optometric clinical practice.