Developing shared understandings of standards-based assessment : online moderation practices across geographically diverse contexts

This thesis investigates the place of online moderation in supporting teachers to work in a system of standards-based assessment. The participants of the study were fifty middle school teachers who met online with the aim of developing consistency in their judgement decisions. Data were gathered through observation of the online meetings, interviews, surveys and the collection of artefacts. The data were viewed and analysed through sociocultural theories of learning and sociocultural theories of technology, and demonstrates how utilising these theories can add depth to understanding the added complexity of developing shared meaning of standards in an online context. The findings contribute to current understanding of standards-based assessment by examining the social moderation process as it acts to increase the reliability of judgements that are made within a standards framework. Specifically, the study investigates the opportunities afforded by conducting social moderation practices in a synchronous online context. The study explicates how the technology affects the negotiation of judgements and the development of shared meanings of assessment standards, while demonstrating how involvement in online moderation discussions can support teachers to become and belong within a practice of standards-based assessment. This research responds to a growing international interest in standards-based assessment and the use of social moderation to develop consistency in judgement decisions. Online moderation is a new practice to address these concerns on a systemic basis.

Sustaining teacher professionalism in the context of standards referenced assessment reform

Standards referenced reform, tied to reporting, engages directly with assessment issues related to accountability. Assessment is the key to good education and is inseparable from curriculum. In an accountability context, standards are used as a lever to improve the reliability and consistency of teacher judgement; and classroom evidence is used by education systems for reporting and tracking achievement over time. Assessment is thus a powerful driver for change and is at the heart of the teaching-learning dynamic. The relationship between the learner, learning and assessment needs to be kept central and the idea of teacher empowerment is fundamental. This chapter is a call to honour and sustain teacher professionalism through educative forms of school-based and teacher-led evaluation, assessment and communities of judgement practice. It supports the argument for a central place for classroom assessment in the role of assessment in educational accountability...

Faculty mentoring, evidence-based assessment, and student learning : an Australian exploration of American initiatives

There have been many improvements in Australian engineering education since the 1990s. However, given the recent drive for assuring the achievement of identified academic standards, more progress needs to be made, particularly in the area of evidence-based assessment. This paper reports on initiatives gathered from the literature and engineering academics in the USA, through an Australian National Teaching Fellowship program. The program aims to establish a process to help academics in designing and implementing evidence-based assessments that meet the needs of not only students and the staff that teach them, but also industry as well as accreditation bodies. The paper also examines the kinds and levels of support necessary for engineering academics, especially early career ones, to help meet the expectations of the current drive for assured quality and standards of both research and teaching. Academics are experiencing competing demands on their time and energy with very high expectations in research performance and increased teaching responsibilities, although many are researchers who have not had much pedagogic training. Based on the literature and investigation of relevant initiatives in the USA, we conducted interviews with several identified experts and change agents who have wrought effective academic cultural change within their institutions and beyond. These reveal that assuring the standards and quality of student learning outcomes through evidence-based assessments cannot be appropriately addressed without also addressing the issue of pedagogic training for academic staff. To be sustainable, such training needs to be complemented by a culture of on-going mentoring support from senior academics, formalised through the university administration, so that mentors are afforded resources, time, and appropriate recognition.

The relationship of aptitude, course grades, and program length, to performance on a standards based test (NCLEX -RN)

The purpose of this study was threefold: first, to investigate variables associated with learning, and performance as measured by the National Council Licensure Examination for Registered Nurses (NCLEX-RN). The second purpose was to validate the predictive value of the Assessment Technologies Institute (ATI) achievement exit exam, and lastly, to provide a model that could be used to predict performance on the NCLEX-RN, with implications for admission and curriculum development. The study was based on school learning theory, which implies that acquisition in school learning is a function of aptitude (pre-admission measures), opportunity to learn, and quality of instruction (program measures). Data utilized were from 298 graduates of an associate degree nursing program in the Southeastern United States. Of the 298 graduates, 142 were Hispanic, 87 were Black, non-Hispanic, 54 White, non-Hispanic, and 15 reported as Others. The graduates took the NCLEX-RN for the first time during the years 2003–2005. This study was a predictive, correlational design that relied upon retrospective data. Point biserial correlations, and chi-square analyses were used to investigate relationships between 19 selected predictor variables and the dichotomous criterion variable, NCLEX-RN. The correlation and chi square findings indicated that men did better on the NCLEX-RN than women; Blacks had the highest failure rates, followed by Hispanics; older students were more likely to pass the exam than younger students; and students who passed the exam started and completed the nursing program with a higher grade point average, than those who failed the exam. Using logistic regression, five statistical models that used variables associated with learning and student performance on the NCLEX-RN were tested with a model adapted from Bloom's (1976) and Carroll's (1963) school learning theories. The derived model included: NCLEX-RNsuccess = f (Nurse Entrance Test and advanced medical-surgical nursing course grade achieved). The model demonstrates that student performance on the NCLEX-RN can be predicted by one pre-admission measure, and a program measure. The Assessment Technologies Institute achievement exit exam (an outcome measure) had no predictive value for student performance on the NCLEX-RN. The model developed accurately predicted 94% of the student's successful performance on the NCLEX-RN.

The impact of standards -based practices in mathematics on the achievement of low -performing students

This study examined standards-based mathematics reform initiatives to determine if they would improve student achievement on the part of low-performing students. New curricula, the Carnegie Learning Cognitive Tutor®, were provided for algebra and geometry students. The new instructional strategy relied on both the teacher-led instruction and the use of computers to differentiate instruction for individual students. Mathematics teachers received ongoing professional development to help them implement the new curricula. In addition, teachers were provided with ongoing support to assist them with the transformation of the learning environments for students using standards-based practices. This quasi-experimental (nonrandomized) study involved teachers in two matched urban high schools. Analyses (ANCOVAs) revealed that the experimental group with an appropriately implemented program had significantly higher learning gains than the comparison group as determined by the students' 2007 mathematics Developmental Scale Score (DSS). In addition, the experimental group's adjusted mean for the second interim mathematics assessment was significantly higher than the comparison group's mean. The findings support the idea that if the traditional curriculum is replaced with standards-based curriculum, and the curriculum is implemented as intended, low-performing students may make significant learning gains. With respect to the teaching practices as observed with the Classroom Observation Protocol (COP), t-tests were conducted on four constructs. The results for both the algebra and geometry teachers on the constructs were not significant. The COP indicated that teachers in both the experimental and comparison groups used traditional instruction strategies in their classrooms. The analyses of covariance (ANCOVA) on the use of technology revealed no significant main effects for computer use.

The impact of standards-based practices in mathematics on the achievement of low-performing students

This study examined standards-based mathematics reform initiatives to determine if they would improve student achievement on the part of low-performing students. New curricula, the Carnegie Learning Cognitive Tutor®, were provided for algebra and geometry students. The new instructional strategy relied on both the teacher-led instruction and the use of computers to differentiate instruction for individual students. Mathematics teachers received ongoing professional development to help them implement the new curricula. In addition, teachers were provided with ongoing support to assist them with the transformation of the learning environments for students using standards-based practices. This quasi-experimental (nonrandomized) study involved teachers in two matched urban high schools. Analyses (ANCOVAs) revealed that the experimental group with an appropriately implemented program had significantly higher learning gains than the comparison group as determined by the students' 2007 mathematics Developmental Scale Score (DSS). In addition, the experimental group's adjusted mean for the second interim mathematics assessment was significantly higher than the comparison group's mean. The findings support the idea that if the traditional curriculum is replaced with standards-based curriculum, and the curriculum is implemented as intended, low-performing students may make significant learning gains. With respect to the teaching practices as observed with the Classroom Observation Protocol (COP), t-tests were conducted on four constructs. The results for both the algebra and geometry teachers on the constructs were not significant. The COP indicated that teachers in both the experimental and comparison groups used traditional instruction strategies in their classrooms. The analyses of covariance (ANCOVA) on the use of technology revealed no significant main effects for computer use.

Education for all : providing curriculum for students with disabilities in a standards-based environment : implications for Queensland schools

This report investigates lessons learned by educators in the United States when providing a standards-based curriculum for all students including Students with Disabilities (SWD). Assumptions about implementation of these lessons are then made to the Queensland school system. Queensland mainstream schools currently provide a standards-based curriculum for over sixteen thousand-four hundred students with mild-moderate disabilities and appear to be challenged by this new educational reform and its implications to school and teacher practices, beliefs and attitudes. The analysis of US research, literature and educational policy for this report, has provided some implications for Queensland schools in the areas of student participation, achievement and curriculum planning to provide an “education for all”. The analysis and comparison of legislation and policy, which demonstrates some significant similarities, provides greater validity for the application of lessons learned in the United States to the Queensland context. The key findings about lessons learned provides Queensland schools with some assumptions as to why and how they need to refocus school leader and teachers’ practices, beliefs and attitudes to provide an “education for all”. These lessons infer that school leaders and teachers to explicitly focus on equity, expectation, accountability, performance, alignment and collaboration so that effective curriculum is provided for SWD, indeed all students, in the Queensland standards-based curriculum environment.

Changing assessment practices : the case for online moderation

Online moderation meetings have the potential to support the collaborative professional development of teachers, and the formation of a common understanding of what denotes quality in student work in a standards based assessment system. In doing so systemic calls for consistency across education systems are also being met. In this paper a case for employing online moderation meetings is developed through recourse to the demands of learning in the twenty-first century and the place of assessment within those discourses. It is argued that empirical data is needed on the efficacy of online moderation meetings to guide future practice as the use of information and communication technologies increases in education systems. Online moderation is one way of gathering teachers across vast distances to share their understandings and develop common meanings of assessment. While it is suggested that online moderation is one possible procedure to meet systemic requirements and support teachers’ professional collaboration, the implementation of such a system also introduces new challenges for schools and teachers. Meeting online to discuss professional understandings is a new way of operating for teachers and involves technology that has not yet been fully utilised within education departments. Issues such as the types of interactions that are afforded within such an environment, as well as technical operating problems that occur when using technology impact on the employment of online meetings. Online moderation meetings while potentially solving the issue of developing common understandings across an entire department also pose new issues to be resolved. There is a need for research into the efficacy of online moderation meetings so that future policy decisions may be based on sound empirical data. It is imperative that as new ways of knowing and acting are incorporated into school curriculum and pedagogy, assessment practices are also aligned. Online moderation meetings can support such practices by enabling teachers to communicate with a wider and more diverse group of teachers to establish common understandings.

Open standards-based system integration for asset management decision support

Over the last decade, system integration has grown in popularity as it allows organisations to streamline business processes. Traditionally, system integration has been conducted through point-to-point solutions – as a new integration scenario requirement arises, a custom solution is built between the relevant systems. Bus-based solutions are now preferred, whereby all systems communicate via an intermediary system such as an enterprise service bus, using a common data exchange model. This research investigates the use of a common data exchange model based on open standards, specifically MIMOSA OSA-EAI, for asset management system integration. A case study is conducted that involves the integration of processes between a SCADA, maintenance decision support and work management system. A diverse number of software platforms are employed in developing the final solution, all tied together through MIMOSA OSA-EAI-based XML web services. The lessons learned from the exercise are presented throughout the paper.

Standards-driven reform years 1-10 : moderation an optional extra?

While externally moderated standards-based assessment has been practised in Queensland senior schooling for more than three decades, there has been no such practice in the middle years. With the introduction of standards at state and national levels in these years, teacher judgement as developed in moderation practices is now vital. This paper argues, that in this context of assessment reform, standards intended to inform teacher judgement and to build assessment capacity are necessary but not sufficient for maintaining teacher and public confidence in schooling. Teacher judgement is intrinsic to moderation, and to professional practice, and can no longer remain private. Moderation too is intrinsic to efforts by the profession to realise judgements that are defensible, dependable and open to scrutiny. Moderation can no longer be considered an optional extra and requires system-level support especially if, as intended, the standards are linked to system-wide efforts to improve student learning. In presenting this argument we draw on an Australian Research Council funded study with key industry partners (the Queensland Studies Authority and the National Council for Curriculum and Assessment of the Republic of Ireland). The data analysed included teacher interview data and additional teacher talk during moderation sessions. These were undertaken during the initial phase of policy development. The analysis identified those issues that emerge in moderation meetings that are designed to reach consistent, reliable judgements. Of interest are the different ways in which teachers talked through and interacted with one another to reach agreement about the quality of student work in the application of standards. There is evidence of differences in the way that teachers made compensations and trade-offs in their award of grades, dependent on the subject domain in which they teach. This article concludes with some empirically derived insights into moderation practices as policy and social events.

The development of teacher assessment identity through participation in online moderation

Teachers’ professional conversations regarding the qualities evidenced in student work provide opportunities to develop a shared understanding of achievement standards. This research investigates social moderation conducted in a synchronous online mode as a specific form of professional conversation. The discussion considers the different factors that influenced these conversations which included the technologic medium of the meeting. The focus of the discussion is how participation in online moderation can support teachers to develop an assessment identity as one who works within a standards-based assessment system. Qualitative data were gathered from middle school teachers from different year levels, in different curriculum areas, in diverse geographic locations, and in a range of sociocultural contexts within Queensland, Australia. Analysis of the data through a sociocultural lens of becoming suggests that participation in online moderation, while challenging for teachers, can also provide opportunities to construct and to negotiate an identity as an assessor of student work.