Moderation as judgement practice: Reconciling system level accountability and local level practice

Internationally, recognition of the role of assessment to inform the learning process has received much attention in recent years. Assessment for learning, not just of learning is being supported by an increasing body of literature providing strategies that teachers and their students can incorporate to support the learning process (Assessment Reform Group, 2002; Broadfoot & Black, 2004; James, 2006). Concurrently there has been an increase internationally in systemic accountability requirements of schools in terms of student results. The convergence of these two movements has resulted in some education systems promoting standards-driven reform involving authentic assessment and a re-examination of the relationship between the teacher and the student in the learning process. In this context standards are intended to be used as the basis for judgements of student achievement; while the results from assessment tasks are meant to both inform the teaching/learning process, and to report and track student progress. In such system, the role and reliability of teacher judgement takes centre stage.

Standards, teacher judgement and moderation in contexts of national curriculum and assessment reform

This paper puts forward a proposal for reviewing the role and purpose of standards in the context of national curriculum and assessment reform more generally. It seeks to commence the much-needed conversation about standards in the work of teachers as distinct from large-scale testing companies and the policy personnel responsible for reporting. Four key conditions that relate to the effective use of standards to measure improvement and support learning are analysed: clarity about purpose and function; understanding of the representation of standards; moderation practice; and the assessment community. The Queensland experience of the use of standards, teacher judgement and moderation is offered to identify what is educationally preferable in terms of their use and their relationships to curriculum, improvement and accountability. The article illustrates how these practices have recently been challenged by emerging political constraints related to the Australian Government’s implementation of national testing and national partnership funding arrangements tied to the performance of students at or below minimum standards.

The centrality of teachers’ judgement practice in assessment : a study of standards in moderation

There is a strong quest in several countries including Australia for greater national consistency in education and intensifying interest in standards for reporting. Given this, it is important to make explicit the intended and unintended consequences of assessment reform strategies and the pressures to pervert and conform. In a policy context that values standardisation, the great danger is that the technical, rationalist approaches that generalise and make superficial assessment practices, will emerge. In this article, the authors contend that the centrality and complexity of teacher judgement practice in such a policy context need to be understood. To this end, we discuss and analyse recorded talk in teacher moderation meetings showing the processes that teachers use as they work with stated standards to award grades (A to E). We show how they move to and fro between (1) supplied textual artefacts, including stated standards and samples of student responses, (2) tacit knowledge of different types, drawing into the moderation, and (3) social processes of dialogue and negotiation. While the stated standards play a part in judgement processes, in and of themselves they are shown to be insufficient to account for how the teachers ascribe value and award a grade to student work in moderation. At issue is the nature of judgement as cognitive and social practice in moderation and the legitimacy (or otherwise) of the mix of factors that shape how judgement occurs.

The role of emotional intelligence in moral judgement

In this study we investigated the potential role of emotional intelligence (EI) in moral reasoning (MR). A sample of 131 undergraduate students completed a battery of psychological tests, which included measures of EI, MR and the Big Five dimensions of personality. Results revealed support for a proposed model of the relationship between emotional intelligence, personality and moral reasoning. Specifically, emotional intelligence was found to be a significant predictor of four of the Big Five personality dimensions (extraversion, openness, neuroticism, agreeableness), which in turn were significant predictors of moral reasoning. These results have important implications in regards to our current understanding of the relationships between EI, moral reasoning and personality. We emphasise the need to incorporate the constructs of EI and moral reasoning into a broader, explanatory personality framework.

Moderation and consistency of teacher judgement : teachers’ views

Major curriculum and assessment reforms in Australia have generated research interest in issues related to standards, teacher judgement and moderation. This article is based on one related inquiry of a large-scale Australian Research Council Linkage project conducted in Queensland. This qualitative study analysed interview data to identify teachers’ views on standards and moderation as a means to achieving consistency of teacher judgement. A complementary aspect of the research involved a blind review that was conducted to determine the degree of teacher consistency without the experience of moderation. Empirical evidence was gained that most teachers, of the total interviewed, articulated a positive attitude towards the use of standards in moderation and perceived that this process produces consistency in teachers’ judgements. Context was identified as an important influential factor in teachers’ judgements and it was concluded that teachers’ assessment beliefs, attitudes and practices impact on their perceptions of the value of moderation practice and the extent to which consistency can be achieved.

Towards an understanding of teacher judgement in the context of social moderation

Social moderation involves teachers gathering together to discuss their judgements of the quality of student work and to reach agreement regarding the standard awarded. This qualitative study conducted over a three-year period investigated the social practice of moderation and the influence on teachers’ judgements of students work. An initial survey of teachers’ understandings of moderation and standards, pre-interviews of teachers who participated in the moderation meetings, observations of these meetings with a particular focus on one teacher (focus teachers) comprised the data collection methods. Data analysis involved organising, matching, coding, identifying patterns and themes using a constant comparative method. Socio-cultural theories of learning and assessment underpinned the approach to data analysis and proved helpful in explaining the diverse influences on teachers’ judgements beyond the task criteria, and the progressive development of shared understandings through engaging in professional discussions of students’ work. The study revealed that the process is not clear and linear and is influenced by factors such as the representation of the standards and the knowledge base of the teachers.

Automata modeling for cognitive interference in users' relevance judgement

Quantum theory has recently been employed to further advance the theory of information retrieval (IR). A challenging research topic is to investigate the so called quantum-like interference in users’ relevance judgement process, where users are involved to judge the relevance degree of each document with respect to a given query. In this process, users’ relevance judgement for the current document is often interfered by the judgement for previous documents, due to the interference on users’ cognitive status. Research from cognitive science has demonstrated some initial evidence of quantum-like cognitive interference in human decision making, which underpins the user’s relevance judgement process. This motivates us to model such cognitive interference in the relevance judgement process, which in our belief will lead to a better modeling and explanation of user behaviors in relevance judgement process for IR and eventually lead to more user-centric IR models. In this paper, we propose to use probabilistic automaton(PA) and quantum finite automaton (QFA), which are suitable to represent the transition of user judgement states, to dynamically model the cognitive interference when the user is judging a list of documents.

A Systems-level approach to building sustainable assessment cultures : moderation, quality task design and dependability of judgement

This paper considers the conditions that are necessary at system and local levels for teacher assessment to be valid, reliable and rigorous. With sustainable assessment cultures as a goal, the paper examines how education systems can support local level efforts for quality learning and dependable teacher assessment. This is achieved through discussion of relevant research and consideration of a case study involving an evaluation of a cross-sectoral approach to promoting confidence in school-based assessment in Queensland, Australia. Building on the reported case study, essential characteristics for developing sustainable assessment cultures are presented, including: leadership in learning; alignment of curriculum, pedagogy and assessment; the design of quality assessment tasks and accompanying standards, and evidence-based judgement and moderation. Taken together, these elements constitute a new framework for building assessment capabilities and promoting quality assurance.

Explicit, latent and meta-criteria : types of criteria at play in professional judgement practice

This paper engages with debates about whether comprehensive prior specification of criteria and standards is sufficient for informed professional judgement. A preoccupation has emerged with the specificity and explication of criteria intended to regulate judgement. This has resulted in criteria-compliance in the use of defined standards to validate judgements and improve reliability and consistency. Compliance has become a priority, the consequence being the prominence of explicit criteria, to the lack of acknowledgement of the operation of latent and meta-criteria within judgement practice. This paper examines judgement as a process involving three categories of assessment criteria in the context of standards-referenced systems: explicit, latent and meta-criteria. These are understood to be wholly interrelated and interdependent. A conceptualisation of judgement involving the interplay of the three criteria types is presented with an exploration of how they function to focus or alter assessments of quality in judgements of achievement in English and Mathematics.

Semantic judgement of medical concepts : combining syntagmatic and paradigmatic information with the tensor encoding model

This paper outlines a novel approach for modelling semantic relationships within medical documents. Medical terminologies contain a rich source of semantic information critical to a number of techniques in medical informatics, including medical information retrieval. Recent research suggests that corpus-driven approaches are effective at automatically capturing semantic similarities between medical concepts, thus making them an attractive option for accessing semantic information. Most previous corpus-driven methods only considered syntagmatic associations. In this paper, we adapt a recent approach that explicitly models both syntagmatic and paradigmatic associations. We show that the implicit similarity between certain medical concepts can only be modelled using paradigmatic associations. In addition, the inclusion of both types of associations overcomes the sensitivity to the training corpus experienced by previous approaches, making our method both more effective and more robust. This finding may have implications for researchers in the area of medical information retrieval.

Investigating the complexity of judgement practice

Underlying all assessments are human judgements regarding the quality of students’ understandings. Despite their ubiquity, those judgements are conceptually elusive. The articles selected for inclusion in this issue explore the complexity of judgement practice raising critical questions that challenge existing views and accepted policy and practice.

Towards improving public understanding of judgement practice in standards-referenced assessment : an Australian perspective

Internationally there has been a move towards standards-referenced assessment with countries such as Australia developing a National Curriculum and Achievement Standards, New Zealand adopting National Standards for literacy and numeracy that involve schools making and reporting judgements about the reading, writing and mathematics achievement of children up to Year 8 (the end of primary school) and in Canada, classroom assessment standards aimed at the improvement of assessment practice of K-12 education are being formulated. Standards-driven reform has major implications for teachers’ work. The consequences of adopting a standards-driven approach to educational change by systems are often under-estimated with the unintended effects not fully understood by either the policy writers, and the public, including parents. It is for these reasons that the contention developed in this article relates to the teacher’s role, which it is argued remains central to policy focused on the improvement of the quality of education and educational standards.

Board monitoring and judgement as processes of sensemaking

When organizational scandals occur, the common refrain among commentators is: 'Where was the board in all this?' 'How could the directors not have known what was going on?''Why didn't the board intervene?' The scandals demonstrate that board monitoring or oganizational performance is a matter of great importance. By monitoring, we mean the act of keeping the organization under review. In many English-speaking countries, directors have a legal duty of care, which includes duties to monitor the performance of their organizations (Hopt and von Hippel 2010). However, statutory law typically merely states the duty, while providing little guidance on how that duty can be met.

Assessment for Education : Standards, Judgement and Moderation

Do you need a practical guide to assessment, curriculum and policy? Are you also looking for a book that is firmly grounded in the theory of this subject? Assessment for Education combines both theory and practice, making it the perfect guide for students, researchers, academics and teachers. This book makes assessment processes transparent for practitioners, and shows how assessment should relate to education. It looks at evidence-informed decision-making and the interrelationships between standards, judgment and moderation practice for improved assessment, teacher quality, schools and systems. The book will provide you with: ' Knowledge about quality assessment and judgement practice ' Understanding of relationships across curriculum, assessment, teaching and learning ' Knowledge of the concept of front-ending assessment based on the learner's needs ' An analysis of practitioner judgement approaches ' Understanding of the conditions under which teacher assessment can be valid ' Principles derived from research of social moderation practices Whether you are studying and researching assessment or working in curriculum and assessment policy, this book will show you how practitioner use of achievement standards can improve learning, equity, social justice and accountability.