Changing assessment practices : the case for online moderation

Online moderation meetings have the potential to support the collaborative professional development of teachers, and the formation of a common understanding of what denotes quality in student work in a standards based assessment system. In doing so systemic calls for consistency across education systems are also being met. In this paper a case for employing online moderation meetings is developed through recourse to the demands of learning in the twenty-first century and the place of assessment within those discourses. It is argued that empirical data is needed on the efficacy of online moderation meetings to guide future practice as the use of information and communication technologies increases in education systems. Online moderation is one way of gathering teachers across vast distances to share their understandings and develop common meanings of assessment. While it is suggested that online moderation is one possible procedure to meet systemic requirements and support teachers’ professional collaboration, the implementation of such a system also introduces new challenges for schools and teachers. Meeting online to discuss professional understandings is a new way of operating for teachers and involves technology that has not yet been fully utilised within education departments. Issues such as the types of interactions that are afforded within such an environment, as well as technical operating problems that occur when using technology impact on the employment of online meetings. Online moderation meetings while potentially solving the issue of developing common understandings across an entire department also pose new issues to be resolved. There is a need for research into the efficacy of online moderation meetings so that future policy decisions may be based on sound empirical data. It is imperative that as new ways of knowing and acting are incorporated into school curriculum and pedagogy, assessment practices are also aligned. Online moderation meetings can support such practices by enabling teachers to communicate with a wider and more diverse group of teachers to establish common understandings.

Developing shared understandings of standards-based assessment : online moderation practices across geographically diverse contexts

This thesis investigates the place of online moderation in supporting teachers to work in a system of standards-based assessment. The participants of the study were fifty middle school teachers who met online with the aim of developing consistency in their judgement decisions. Data were gathered through observation of the online meetings, interviews, surveys and the collection of artefacts. The data were viewed and analysed through sociocultural theories of learning and sociocultural theories of technology, and demonstrates how utilising these theories can add depth to understanding the added complexity of developing shared meaning of standards in an online context. The findings contribute to current understanding of standards-based assessment by examining the social moderation process as it acts to increase the reliability of judgements that are made within a standards framework. Specifically, the study investigates the opportunities afforded by conducting social moderation practices in a synchronous online context. The study explicates how the technology affects the negotiation of judgements and the development of shared meanings of assessment standards, while demonstrating how involvement in online moderation discussions can support teachers to become and belong within a practice of standards-based assessment. This research responds to a growing international interest in standards-based assessment and the use of social moderation to develop consistency in judgement decisions. Online moderation is a new practice to address these concerns on a systemic basis.

An investigation into online moderation

The purpose of this paper is to understand how teachers’ identity as an assessor in a standards-referenced assessment system may be developed through their participation in online social moderation meetings. In these meetings teachers negotiate and share their understandings of assessment standards and judgement decisions. In particular, the paper focuses on the relationship between the technology, the moderation processes and teachers’ development in this assessment system. This paper draws on sociocultural theories of learning to analyse the qualitative data collected through observations of eleven online moderation meetings and interviews of the teachers involved in these meetings. The results provide insights into the mediating role of the technology with regard to teachers’ development of shared meanings and common practices within a standards-referenced assessment system.

Learning as identity and practice through involvement in online moderation

For teachers working in a standards-based assessment system, professional conversations through organised social moderation meetings are a vital element. This qualitative research investigated the learning that occurred as a result of online moderation discussions. Findings illustrate how participating in social moderation meetings in an online context can support teachers to understand themselves as assessors, and can provide opportunities for teachers to imagine possibilities for their teaching that move beyond the moderation practice.

The development of teacher assessment identity through participation in online moderation

Teachers’ professional conversations regarding the qualities evidenced in student work provide opportunities to develop a shared understanding of achievement standards. This research investigates social moderation conducted in a synchronous online mode as a specific form of professional conversation. The discussion considers the different factors that influenced these conversations which included the technologic medium of the meeting. The focus of the discussion is how participation in online moderation can support teachers to develop an assessment identity as one who works within a standards-based assessment system. Qualitative data were gathered from middle school teachers from different year levels, in different curriculum areas, in diverse geographic locations, and in a range of sociocultural contexts within Queensland, Australia. Analysis of the data through a sociocultural lens of becoming suggests that participation in online moderation, while challenging for teachers, can also provide opportunities to construct and to negotiate an identity as an assessor of student work.

Reflections, successes and pitfalls of coordinating online discussion forums in a foreign language

The advent of e-learning has seen the adaptation and use of a plethora of educational techniques. Of these, online discussion forums have met with success and been used widely in both undergraduate and postgraduate education. The authors of this paper, having previously used online discussion forums in the postgraduate arena with success, adopted this approach for the design and subsequent delivery of a learning and teaching subject. This learning and teaching subject, however, was part of an international collaboration and designed for nurse academics in another country – Vietnam. With the nursing curriculum in Vietnam currently moving to adopt a competency based approach, two learning and teaching subjects were designed by an Australian university for Vietnamese nurse academics. Subject materials constituted a DVD which arrived by post and access to an online platform. Assessment for the subject included (but was not limited to) mandatory participation in online discussion with the other nurse academics enrolled in the subject. The purpose behind the online discussion was to generate discourse between the Vietnamese nurse academics located across Vietnam. Consequently the online discussions occurred in both Vietnamese and English; the Australian academic moderating the discussion did so in Australia with a Vietnamese translator. For the Australian University delivering this subject the difference between this and past online discussions were twofold: delivery was in a foreign language; and the teaching experience of the Vietnamese nurse teachers was mixed and frequently very limited. This paper will provide a discussion addressing the design of an online learning environment for foreign correspondents, the resources and translation required to maximise the success of the online discussion, the lessons learnt and consequent changes made, as well as the rationale of delivering complex content in a foreign language. While specifically addressing the first iteration of the first learning module designed, this paper will also address subsequent changes made for the second iteration of the first module and comment on their success. While a translator is clearly a key component of success, the elements of simplicity and clarity in hand with supportive online moderation must not be overlooked.

Bilingual asynchronous online discussion groups : design and delivery of an e-Learning distance study module for nurse academics in a developing country

The advent of eLearning has seen online discussion forums widely used in both undergraduate and postgraduate nursing education. This paper reports an Australian university experience of design, delivery and redevelopment of a distance education module developed for Vietnamese nurse academics. The teaching experience of Vietnamese nurse academics is mixed and frequently limited. It was decided that the distance module should attempt to utilise the experience of senior Vietnamese nurse academics - asynchronous online discussion groups were used to facilitate this. Online discussion occurred in both Vietnamese and English and was moderated by an Australian academic working alongside a Vietnamese translator. This paper will discuss the design of an online learning environment for foreign correspondents, the resources and translation required to maximise the success of asynchronous online discussion groups, as well as the rationale of delivering complex content in a foreign language. While specifically addressing the first iteration of the first distance module designed, this paper will also address subsequent changes made for the second iteration of the module and comment on their success. While a translator is clearly a key component of success, the elements of simplicity and clarity combined with supportive online moderation must not be overlooked.

Towards a theoretical framework for online professional discussions

Online interactions are becoming commonplace for a multitude of educational purposes. Each context presents a unique and dynamic mix of variables that combine to shape the practice and the identities of those involved. In this article, sociocultural theories of learning and sociocultural theories of technology are explored as a way to view and to map the complex interactions that can occur. The case of synchronous online moderation meetings are used as an example of the combination of variables that can impact on the development of shared understandings of a practice. Online moderation can involve teachers from geographically diverse areas discussing and negotiating their judgement decisions. These discussions represent an intersection of a national curriculum, standards-referenced assessment, moderation protocols, site-specific practices and understandings, and individual teachers’ knowledges and histories. It is suggested that the proposed theoretical combination addresses some of the limitations of each of the theories when investigating such a dynamic context. As higher education moves into increasing use of online modes of communication and a higher level of accountability the relevance of this discussion to higher education is evident.

The development of shared understandings of assessment policy : travelling between global and local contexts

In this paper, teachers’ enactment of assessment policy within demands for accountability and consistency of teacher judgements is considered. Evidence is drawn from a qualitative study involving 50 middle school teachers from Queensland, Australia, who participated in online social moderation meetings with teachers located in dispersed areas around the state. The study presents how travelling policy is embedded in local histories and cultures, in particular within systems of accountability; and the different layers of what may be considered ‘local’. The paper examines the intersections of travelling and embedded policy, and global and local contexts as these are enacted through online moderation meetings.

Reconceptualising moderation in asynchronous online discussions using grounded theory.

This paper reports a grounded theory study of e-moderation of asynchronous online discussions, to explore the processes by which tutors in higher education decide when and how to e-moderate. It aims to construct a theory of e-moderation based on some key factors which appear to influence e-moderation. It discusses previous research on the definition and practice of e-moderation, and then describes the study, which involved four e-moderators working in two different university contexts. Key themes on e-moderation, which emerged using a grounded theory approach, are discussed. It proposes a paradigm framework for e-moderation and suggests that as a facilitative activity, it should be sufficiently contained within ‘a ring-fenced learning arena’. Factors outwith and inside the ring-fence which appear to influence e-moderation and their implications for future theory development and validation are discussed. Keywords: e-moderation; grounded theory; online learning; asynchronous discussions; ring-fence

Developing consistency in teacher judgment formation through online meeting centres

This paper investigates the complex interactions that occur as teachers meet online to justify and negotiate their assessment judgments of student work across relatively large and geographically dispersed populations. Drawing from sociocultural theories of learning and technology, the technology is positioned as playing a role in either supporting or hindering teachers reaching a common understanding of assessment standards. Meeting transcripts and interviews with the teachers have been qualitatively analysed in terms of the interactions that occurred and teachers’ perceptions of these interactions. While online meetings offer a partial solution to address the current demands of assessment in education, they also present new challenges as teachers meet, in an unfamiliar environment, to discuss student work.

Standards-driven reform years 1-10 : moderation an optional extra?

While externally moderated standards-based assessment has been practised in Queensland senior schooling for more than three decades, there has been no such practice in the middle years. With the introduction of standards at state and national levels in these years, teacher judgement as developed in moderation practices is now vital. This paper argues, that in this context of assessment reform, standards intended to inform teacher judgement and to build assessment capacity are necessary but not sufficient for maintaining teacher and public confidence in schooling. Teacher judgement is intrinsic to moderation, and to professional practice, and can no longer remain private. Moderation too is intrinsic to efforts by the profession to realise judgements that are defensible, dependable and open to scrutiny. Moderation can no longer be considered an optional extra and requires system-level support especially if, as intended, the standards are linked to system-wide efforts to improve student learning. In presenting this argument we draw on an Australian Research Council funded study with key industry partners (the Queensland Studies Authority and the National Council for Curriculum and Assessment of the Republic of Ireland). The data analysed included teacher interview data and additional teacher talk during moderation sessions. These were undertaken during the initial phase of policy development. The analysis identified those issues that emerge in moderation meetings that are designed to reach consistent, reliable judgements. Of interest are the different ways in which teachers talked through and interacted with one another to reach agreement about the quality of student work in the application of standards. There is evidence of differences in the way that teachers made compensations and trade-offs in their award of grades, dependent on the subject domain in which they teach. This article concludes with some empirically derived insights into moderation practices as policy and social events.

Building engaged, sustainable online communities : a case study of an adventure travel website

This paper will provide an overview of a join research initiative being developed by the Queensland University of Technology in conjunction with the Australian Smart Services Cooperative Research Centre in relation to the development and analysis of online communities. The intention of this project is to initially create an exciting and innovative web space around the concept of adventure travel and then to analyse the level of user engagement to uncover possible patterns and processes that could be used in the future development of other virtual online communities. Travel websites are not a new concept and there are many successful examples currently operating and generating profit. The intention of the QUT/Smart Services CRC project is to analyse the site metrics to determine the following: what specific conditions/parameters are required to foster a growing and engaged virtual community; when does the shift occur from external moderation to a more sustainable system of self-moderation within the online community; when do users begin to take ownership of a site and take an invested interested in the content and growth of an online community; and how to retain active contributors and high-impact power users on a long-term basis. With the travel website rapidly approaching release, this paper begins the process of reflection, outlining the process undertaken and the findings so far aggregated whilst also positioning the project within the greater context of current online user participation and user generated content research.

Towards an understanding of teacher judgement in the context of social moderation

Social moderation involves teachers gathering together to discuss their judgements of the quality of student work and to reach agreement regarding the standard awarded. This qualitative study conducted over a three-year period investigated the social practice of moderation and the influence on teachers’ judgements of students work. An initial survey of teachers’ understandings of moderation and standards, pre-interviews of teachers who participated in the moderation meetings, observations of these meetings with a particular focus on one teacher (focus teachers) comprised the data collection methods. Data analysis involved organising, matching, coding, identifying patterns and themes using a constant comparative method. Socio-cultural theories of learning and assessment underpinned the approach to data analysis and proved helpful in explaining the diverse influences on teachers’ judgements beyond the task criteria, and the progressive development of shared understandings through engaging in professional discussions of students’ work. The study revealed that the process is not clear and linear and is influenced by factors such as the representation of the standards and the knowledge base of the teachers.

Developing a moderation community of practice

This paper reports on a study to evaluate technology-based processes for assessment moderation. The aim was to evaluate standard features found in an institutional Learning Management System, and their compatibility with the values and practices of a large teaching team. The process used an online discussion board forum for tutors, the paring of more experienced tutors with those new to the process, and further meetings conducted in both face-to-face and web conferencing environments. Online rubrics were used for assessing student work and the provision of feedback. A focus group conducted after marking was concluded and the analysis of the discussion board forum demonstrated a strong community of practice with a shared understanding of assessment requirements.