Assessment for learning in the accountability era : Queensland, Australia

Developments in school education in Australia over the past decade have witnessed the rise of national efforts to reform curriculum, assessment and reporting. Constitutionally the power to decide on curriculum matters still resides with the States. Higher stakes in assessment, brought about by national testing and international comparative analyses of student achievement data, have challenged State efforts to maintain the emphasis on assessment to promote learning while fulfilling accountability demands. In this article lessons from the Queensland experience indicate that it is important to build teachers' assessment capacity and their assessment literacy for the promotion of student learning. It is argued that teacher assessment can be a source of dependable results through moderation practice. The Queensland Studies Authority has recognised and supported the development of teacher assessment and moderation practice in the context of standards-driven, national reform. Recent research findings explain how the focus on learning can be maintained by avoiding an over-interpretation of test results in terms of innate ability and limitations and by encouraging teachers to adopt more tailored diagnosis of assessment data to address equity through focus on achievement for all. Such efforts are challenged as political pressures related to the Australian government’s implementation of national testing and national partnership funding arrangements tied to the performance of students at or below minimum standards become increasingly apparent.

Assessment reform and educational change in Australia

This chapter addresses the changing climate of assessment policy and practice in Australia in response to global trends in education and the mounting accountability demands of standards-driven reform. Queensland, a State of Australia, has a tradition of respecting and trusting teacher judgment through the practice of, and policy commitment to, externally moderated school-based assessment. This chapter outlines the global trends in curriculum and assessment reform, and then analyzes the impact of international comparisons on national policy. The creation of the Australian Curriculum, Assessment and Reporting Authority (ACARA) together with the intent of establishing a standards-referenced framework raises tensions and challenges for teachers’ practice. The argument for sustaining confidence in teacher-based assessment is developed with reference to research evidence pertaining to the use of more authentic assessments and moderation practices for the purposes of improving learning, equity and accountability. Evidence is drawn from local studies of teacher judgment practice and used to demonstrate these developments and in so doing illuminate the complex issues of engaging the demands of policy while sustaining confidence in teacher assessment.

Enterprise information security policy assessment : an extended framework for metrics development utilising the goal-question-metric approach

Effective enterprise information security policy management requires review and assessment activities to ensure information security policies are aligned with business goals and objectives. As security policy management involves the elements of policy development process and the security policy as output, the context for security policy assessment requires goal-based metrics for these two elements. However, the current security management assessment methods only provide checklist types of assessment that are predefined by industry best practices and do not allow for developing specific goal-based metrics. Utilizing theories drawn from literature, this paper proposes the Enterprise Information Security Policy Assessment approach that expands on the Goal-Question-Metric (GQM) approach. The proposed assessment approach is then applied in a case scenario example to illustrate a practical application. It is shown that the proposed framework addresses the requirement for developing assessment metrics and allows for the concurrent undertaking of process-based and product-based assessment. Recommendations for further research activities include the conduct of empirical research to validate the propositions and the practical application of the proposed assessment approach in case studies to provide opportunities to introduce further enhancements to the approach.

A Systems-level approach to building sustainable assessment cultures : moderation, quality task design and dependability of judgement

This paper considers the conditions that are necessary at system and local levels for teacher assessment to be valid, reliable and rigorous. With sustainable assessment cultures as a goal, the paper examines how education systems can support local level efforts for quality learning and dependable teacher assessment. This is achieved through discussion of relevant research and consideration of a case study involving an evaluation of a cross-sectoral approach to promoting confidence in school-based assessment in Queensland, Australia. Building on the reported case study, essential characteristics for developing sustainable assessment cultures are presented, including: leadership in learning; alignment of curriculum, pedagogy and assessment; the design of quality assessment tasks and accompanying standards, and evidence-based judgement and moderation. Taken together, these elements constitute a new framework for building assessment capabilities and promoting quality assurance.

The development of teacher assessment identity through participation in online moderation

Teachers’ professional conversations regarding the qualities evidenced in student work provide opportunities to develop a shared understanding of achievement standards. This research investigates social moderation conducted in a synchronous online mode as a specific form of professional conversation. The discussion considers the different factors that influenced these conversations which included the technologic medium of the meeting. The focus of the discussion is how participation in online moderation can support teachers to develop an assessment identity as one who works within a standards-based assessment system. Qualitative data were gathered from middle school teachers from different year levels, in different curriculum areas, in diverse geographic locations, and in a range of sociocultural contexts within Queensland, Australia. Analysis of the data through a sociocultural lens of becoming suggests that participation in online moderation, while challenging for teachers, can also provide opportunities to construct and to negotiate an identity as an assessor of student work.

Using mobile technology and augmented reality to increase data reliability for environmental assessment

Climate change and land use pressures are making environmental monitoring increasingly important. As environmental health is degrading at an alarming rate, ecologists have tried to tackle the problem by monitoring the composition and condition of environment. However, traditional monitoring methods using experts are manual and expensive; to address this issue government organisations designed a simpler and faster surrogate-based assessment technique for consultants, landholders and ordinary citizens. However, it remains complex, subjective and error prone. This makes collected data difficult to interpret and compare. In this paper we describe a work-in-progress mobile application designed to address these shortcomings through the use of augmented reality and multimedia smartphone technology.

Improving criteria and feedback in student assessment in law

Spurred on by both the 1987 Pearce Report1 and the general changes to higher education spawned by the “Dawkins revolution” from 1988, there has been much critical self-evaluation leading to profound improvements to the quality of teaching in Australian law schools.2 Despite the changes there are still areas of general law teaching practice which have lagged behind recent developments in our understanding of what constitutes high quality teaching. One such area is assessment criteria and feedback. The project Improving Feedback in Student Assessment in Law is an attempt to remedy this. It aims to produce a manual containing key principles for the design of assessment and the provision of feedback, with practical yet flexible ideas and illustrations which law teachers may adopt or modify. Most of the examples have been developed by teachers at the University of Melbourne Law School. The project was supported in 1996 by a Committee for the Advancement of University Teaching grant and the manual will be published late in 1997.3 This note summarises the core principles which are elaborated further in the manual.

Are dietetics educators' attitudes to assessment a barrier to expanding placement opportunities? Results of a Delphi study

Aim Assessment of entry-level health professionals is complex, especially in the work-based setting, placing additional pressures on these learning environments. The present study aims to gain understanding and ideally consensus regarding the setting for assessment of all elements of competence for entry-level dietitians across Australia. Methods Seventy-five experienced academic and practitioner assessors were invited to participate in an online Delphi survey. The 166 entry-level performance criteria of the competency standards for dietitians formed the basis of the questions in the survey, with rating on which ones could be assessed in the practice setting, those which could be assessed in a classroom/university setting and which could be assessed in either setting. Forty-three of 75 invited assessors responded to the first round of the Delphi. A second modified survey was sent to the 43 participants with 34 responding. Results Consensus was achieved for the assessment setting for 86 (52%) of the performance criteria after two rounds of surveying. The majority of these performance criteria achieved consensus at round one (n = 44) and were deemed to be best assessed in the practice setting (n = 55). This study highlighted the perspectives of assessors and their preference for the work-based setting for assessment. Conclusions To reduce the focus on work-based settings as the only place for competence-based assessment of health professionals, there is a need to support individual and organisational change through challenging existing norms around assessment.

Evaluation of functional performance in commercial buildings

This report fully summarises a project designed to enhance commercial real estate performance within both operational and investment contexts through the development of a model aimed at supporting improved decision-making. The model is based on a risk adjusted discounted cash flow, providing a valuable toolkit for building managers, owners, and potential investors for evaluating individual building performance in terms of financial, social and environmental criteria over the complete life-cycle of the asset. The ‘triple bottom line’ approach to the evaluation of commercial property has much significance for the administrators of public property portfolios in particular. It also has applications more generally for the wider real estate industry given that the advent of ‘green’ construction requires new methods for evaluating both new and existing building stocks. The research is unique in that it focuses on the accuracy of the input variables required for the model. These key variables were largely determined by market-based research and an extensive literature review, and have been fine-tuned with extensive testing. In essence, the project has considered probability-based risk analysis techniques that required market-based assessment. The projections listed in the partner engineers’ building audit reports of the four case study buildings were fed into the property evaluation model developed by the research team. The results are strongly consistent with previously existing, less robust evaluation techniques. And importantly, this model pioneers an approach for taking full account of the triple bottom line, establishing a benchmark for related research to follow. The project’s industry partners expressed a high degree of satisfaction with the project outcomes at a recent demonstration seminar. The project in its existing form has not been geared towards commercial applications but it is anticipated that QDPW and other industry partners will benefit greatly by using this tool for the performance evaluation of property assets. The project met the objectives of the original proposal as well as all the specified milestones. The project has been completed within budget and on time. This research project has achieved the objective by establishing research foci on the model structure, the key input variable identification, the drivers of the relevant property markets, the determinants of the key variables (Research Engine no.1), the examination of risk measurement, the incorporation of risk simulation exercises (Research Engine no.2), the importance of both environmental and social factors and, finally the impact of the triple bottom line measures on the asset (Research Engine no. 3).

Application of bag sampling technique for particle size distribution measurements

Bag sampling techniques can be used to temporarily store an aerosol and therefore provide sufficient time to utilize sensitive but slow instrumental techniques for recording detailed particle size distributions. Laboratory based assessment of the method were conducted to examine size dependant deposition loss coefficients for aerosols held in VelostatTM bags conforming to a horizontal cylindrical geometry. Deposition losses of NaCl particles in the range of 10 nm to 160 nm were analysed in relation to the bag size, storage time, and sampling flow rate. Results of this study suggest that the bag sampling method is most useful for moderately short sampling periods of about 5 minutes.

A principal at work : a story of leadership for building sustainable capacity of a school

This study explores through a lifestream narrative how the life experiences of a female primary school principal are organised as practical knowledge, and are used to inform action that is directed towards creating a sustainable school culture. An alternative model of school leadership is presented which describes the thinking and activity of a leader as a process. The process demonstrates how a leader's practical knowledge is dynamic, broadly based in experiential life, and open to change. As such, it is described as a model of sustainable leadership-in-process. The research questions at the heart of this study are: How does a leader construct and organize knowledge in the enactment of the principal ship to deal with the dilemmas and opportunities that arise everyday in school life? And: What does this particular way of organising knowledge look like in the effort to build a sustainable school community? The sustainable leadership-in-process thesis encapsulates new ways of leading primary schools through the principalship. These new ways are described as developing and maintaining the following dimensions of leadership: quality relationships, a collective (shared vision), collaboration and partnerships, and high achieving learning environments. Such dimensions are enacted by the principal through the activities of conversations, performance development, research and data-driven action, promoting innovation, and anticipating and predicting the future. Sustainable leadership-in-process is shared, dynamic, visible and transparent and is conducted through the processes of positioning, defining, organising, experimenting and evaluating in a continuous and iterative way. A rich understanding of the specificity of the life of a female primary school principal was achieved using story telling, story listening and story creation in a collaborative relationship between the researcher and the researched participant. as a means of educational theorising. Analysis and interpretation were undertaken as a recursive process in which the immediate interpretations were shared with the researched participant. The view of theorising adopted in this research is that of theory as hermeneutic; that is, theory is generated out of the stories of experiential life, rather than discovered in the stories.

Standards-driven reform years 1-10 : moderation an optional extra?

While externally moderated standards-based assessment has been practised in Queensland senior schooling for more than three decades, there has been no such practice in the middle years. With the introduction of standards at state and national levels in these years, teacher judgement as developed in moderation practices is now vital. This paper argues, that in this context of assessment reform, standards intended to inform teacher judgement and to build assessment capacity are necessary but not sufficient for maintaining teacher and public confidence in schooling. Teacher judgement is intrinsic to moderation, and to professional practice, and can no longer remain private. Moderation too is intrinsic to efforts by the profession to realise judgements that are defensible, dependable and open to scrutiny. Moderation can no longer be considered an optional extra and requires system-level support especially if, as intended, the standards are linked to system-wide efforts to improve student learning. In presenting this argument we draw on an Australian Research Council funded study with key industry partners (the Queensland Studies Authority and the National Council for Curriculum and Assessment of the Republic of Ireland). The data analysed included teacher interview data and additional teacher talk during moderation sessions. These were undertaken during the initial phase of policy development. The analysis identified those issues that emerge in moderation meetings that are designed to reach consistent, reliable judgements. Of interest are the different ways in which teachers talked through and interacted with one another to reach agreement about the quality of student work in the application of standards. There is evidence of differences in the way that teachers made compensations and trade-offs in their award of grades, dependent on the subject domain in which they teach. This article concludes with some empirically derived insights into moderation practices as policy and social events.

Assessing population limits : a carrying capacity approach to planning

While purporting to enhance Australia’s sustainability, the federal government’s Population Strategy rejects the assessment of the limiting factors to future population growth, thus avoiding urgent threshold issues such as resource depletion and environmental destruction. A more forward-thinking and whole-system perspective would assess and incorporate critical biophysical limits into governance processes with suitable prioritisation. It would encourage communities to examine their individual and collective responsibilities in the context of these limits in order to most equitably optimise outcomes; and it would employ both a resource-based examination of minimum population requirements, and an impact-based assessment of maximum thresholds. This carrying capacity approach to planning could help guide society towards a more sustainable future.

Towards an understanding of teacher judgement in the context of social moderation

Social moderation involves teachers gathering together to discuss their judgements of the quality of student work and to reach agreement regarding the standard awarded. This qualitative study conducted over a three-year period investigated the social practice of moderation and the influence on teachers’ judgements of students work. An initial survey of teachers’ understandings of moderation and standards, pre-interviews of teachers who participated in the moderation meetings, observations of these meetings with a particular focus on one teacher (focus teachers) comprised the data collection methods. Data analysis involved organising, matching, coding, identifying patterns and themes using a constant comparative method. Socio-cultural theories of learning and assessment underpinned the approach to data analysis and proved helpful in explaining the diverse influences on teachers’ judgements beyond the task criteria, and the progressive development of shared understandings through engaging in professional discussions of students’ work. The study revealed that the process is not clear and linear and is influenced by factors such as the representation of the standards and the knowledge base of the teachers.