Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

Assessment for Education : Standards, Judgement and Moderation

Do you need a practical guide to assessment, curriculum and policy? Are you also looking for a book that is firmly grounded in the theory of this subject? Assessment for Education combines both theory and practice, making it the perfect guide for students, researchers, academics and teachers. This book makes assessment processes transparent for practitioners, and shows how assessment should relate to education. It looks at evidence-informed decision-making and the interrelationships between standards, judgment and moderation practice for improved assessment, teacher quality, schools and systems. The book will provide you with: ' Knowledge about quality assessment and judgement practice ' Understanding of relationships across curriculum, assessment, teaching and learning ' Knowledge of the concept of front-ending assessment based on the learner's needs ' An analysis of practitioner judgement approaches ' Understanding of the conditions under which teacher assessment can be valid ' Principles derived from research of social moderation practices Whether you are studying and researching assessment or working in curriculum and assessment policy, this book will show you how practitioner use of achievement standards can improve learning, equity, social justice and accountability.

Literacy assessment that counts : mediating, interpreting and contesting translocal policy in a primary school

In Australia, as in many western education systems over the last two decades, discourses of accountability and performativity have reshaped education policy that has in turn reorganised the work of school leaders and teachers. One of the effects of this reorganisation is increased attention to the production, analysis and display of student achievement data. In this paper we examine in detail a sequence of the production and reading of literacy assessment data in a small Catholic school. Our analysis uses institutional ethnography’s concept of the ‘active text’, the text as occurring in a specific place and time even as it is articulated to social relations beyond its immediate context. Through this process we learn from those involved how their everyday work brings into being formalised, textually authorised processes in a local site that ensure the school meets accountability requirements while enabling teachers to resist standardisation of literacy teaching and assessment.

Blended delivery and online assessment : scaffolding student reflections in work-integrated learning

This paper documents a teaching innovation addressing the challenges of embedding and assessing reflective practice in work-integrated learning, specifically marketing internships. We identify four issues relating to this problem: lack of knowledge or skill for reflection, limitations of physical journals, facilitation of different forms of reflection, and suitable models for teaching and assessing reflection. The paper outlines a blended approach combining face-to-face workshops and online resources, and using online reflective journals and digital stories as assessment. The approach and assessment tasks can be implemented in a variety of marketing and business units.

Implementation of sustainable evidence-based practice for the assessment and management of pain in residential aged care facilities

Pain is common in residential aged care facilities (RACFs). In 2005, the Australian Pain Society developed 27 recommendations for good practice in the identification, assessment, and management of pain in these settings. This study aimed to address implementation of the standards and evaluate outcomes. Five facilities in Australia participated in a comprehensive evaluation of RACF pain practice and outcomes. Pre-existing pain management practices were compared with the 27 recommendations, before an evidence-based pain management program was introduced that included training and education for staff and revised in-house pain-management procedures. Post-implementation audits evaluated the program's success. Aged care staff teams also were assessed on their reports of self-efficacy in pain management. The results show that before the implementation program, the RACFs demonstrated full compliance on 6 to 12 standards. By the project's completion, RACFs demonstrated full compliance with 10 to 23 standards and major improvements toward compliance in the remaining standards. After implementation, the staff also reported better understanding of the standards (p < .001) or of facility pain management guidelines (p < .001), increased confidence in therapies for pain management (p < .001), and increased confidence in their training to assess pain (p < .001) and recognize pain in residents with dementia who are nonverbal (p = .003). The results show that improved evidence-based practice in RACFs can be achieved with appropriate training and education. Investing resources in the aged care workforce via this implementation program has shown improvements in staff self-efficacy and practice.

The Catlin Seaview Survey – kilometre-scale seascape assessment, and monitoring of coral reef ecosystems

1.Marine ecosystems provide critically important goods and services to society, and hence their accelerated degradation underpins an urgent need to take rapid, ambitious and informed decisions regarding their conservation and management. 2.The capacity, however, to generate the detailed field data required to inform conservation planning at appropriate scales is limited by time and resource consuming methods for collecting and analysing field data at the large scales required. 3.The ‘Catlin Seaview Survey’, described here, introduces a novel framework for large-scale monitoring of coral reefs using high-definition underwater imagery collected using customized underwater vehicles in combination with computer vision and machine learning. This enables quantitative and geo-referenced outputs of coral reef features such as habitat types, benthic composition, and structural complexity (rugosity) to be generated across multiple kilometre-scale transects with a spatial resolution ranging from 2 to 6 m2. 4.The novel application of technology described here has enormous potential to contribute to our understanding of coral reefs and associated impacts by underpinning management decisions with kilometre-scale measurements of reef health. 5.Imagery datasets from an initial survey of 500 km of seascape are freely available through an online tool called the Catlin Global Reef Record. Outputs from the image analysis using the technologies described here will be updated on the online repository as work progresses on each dataset. 6.Case studies illustrate the utility of outputs as well as their potential to link to information from remote sensing. The potential implications of the innovative technologies on marine resource management and conservation are also discussed, along with the accuracy and efficiency of the methodologies deployed.

Design, construction and evaluation of systems to predict risk in obstetrics

We present a systematic, practical approach to developing risk prediction systems, suitable for use with large databases of medical information. An important part of this approach is a novel feature selection algorithm which uses the area under the receiver operating characteristic (ROC) curve to measure the expected discriminative power of different sets of predictor variables. We describe this algorithm and use it to select variables to predict risk of a specific adverse pregnancy outcome: failure to progress in labour. Neural network, logistic regression and hierarchical Bayesian risk prediction models are constructed, all of which achieve close to the limit of performance attainable on this prediction task. We show that better prediction performance requires more discriminative clinical information rather than improved modelling techniques. It is also shown that better diagnostic criteria in clinical records would greatly assist the development of systems to predict risk in pregnancy. We present a systematic, practical approach to developing risk prediction systems, suitable for use with large databases of medical information. An important part of this approach is a novel feature selection algorithm which uses the area under the receiver operating characteristic (ROC) curve to measure the expected discriminative power of different sets of predictor variables. We describe this algorithm and use it to select variables to predict risk of a specific adverse pregnancy outcome: failure to progress in labour. Neural network, logistic regression and hierarchical Bayesian risk prediction models are constructed, all of which achieve close to the limit of performance attainable on this prediction task. We show that better prediction performance requires more discriminative clinical information rather than improved modelling techniques. It is also shown that better diagnostic criteria in clinical records would greatly assist the development of systems to predict risk in pregnancy.

Geography and international assessment: Opportunity or distraction

Assessments of students in primary and secondary education are debated among practitioners, policy-makers, and parents. In some countries, assessment scores serve a criterion for passage between levels of education, for example, from secondary school to post-secondary education. Those practices are often traditions and while they come under criticism, they are a long-accepted part of the educational practices within a country. In those countries, the students’ assessment and examination scores are posted in public places or published in local news media. In other countries, assessments are used for the periodic checks on individual student progress. The results of assessments may be used for rating schools, and in some cases, they are used for evaluating the performance of teachers. Assessments are used less often to analyze student performance and make judgments regarding the performance of the curriculum. Even less often, assessments serve to critically establish strategies for the improvement of student learning and educational practices. The ends on the continuum of the assessment debate often focus on the opportunities that assessments present to improve education on one end. The other end is that assessments serve as a major distraction from the important work of teachers by removing classroom room time from instruction. The debate on those issues continues.

Analysis and evaluation of use of the V1 and South East Freeway Bicycleway post completion of Stage C of the V1

Executive Summary: Completion of the Veloway 1 (V1) will provide a dedicated and safe route for cyclists between the Brisbane CBD and the Gateway Motorway off-ramp at Eight Mile Plains alongside the South East Motorway. The V1 is being delivered in stages and when completed will provide a dedicated 3m wide cycleway 17km in length. Two stages (D and E) remain to be constructed to complete the V1. Major trip attractors along the V1 include the Mater, Princes Alexandra and Greenslopes Hospitals, two campuses of Griffith University, Garden City shopping centre and the Australian Tax Office. This report assesses the available evidence on the impacts on cycling behaviour of the recently completed V1 Stage C. The data sources informing this review include three intercept surveys, motion activated traffic cameras and travel time surveys on the V1 and adjoining South East Freeway Bikeway (SEFB), Strava app data, and cyclist crash data along Logan Road. The key findings from the evidence are that the completed V1 Stage C has: a Attracted cyclists from Holland Park, Holland Park West, Mt Gravatt and southern parts of Tarragindi onto the V1 Stage C. b Reduced the crash exposure of pedestrians to cyclists by attracting higher speed cyclists off the adjoining SEFB onto the cycling dedicated V1 Stage C. c Reduced the potential crash exposure of cyclists to motor vehicles by attracting cyclists off Logan Road on to the V1. d Provided travel time benefits to cyclists and reduced road crossings (eight down to two). e Predominantly attracted adults commuting alone to and from work and university. The evidence shows that the two traffic crossings across Birdwood Road (required as a temporary measure until the V1 is completed) negate much of the travel time gains of the V1 Stage C compared to the adjoining SEFB for southbound cyclists. Many cyclists accessing the V1 Stage C from the south are cycling in high-volume vehicular traffic lanes to reduce their travel time along Birdwood Road, but in the process are increasing their exposure to crashes with motor vehicles. Based on these findings this report recommends that TMR: a. Continue with plans to complete the V1 Veloway b. Undertake an engineering feasibility assessment to determine the viability of constructing a section of the V1 Stage E from the intersection Weller and Birdwood Roads over Marshall Road and along Bapaume Road on the western side of the Motorway to the intersection of Bapaume and Sterculia Roads. c. In the interim, improve signage and Birdwood Road crossing points for cyclists accessing and egressing the southern end of the V1 Stage C. d. Work with Brisbane City Council to identify the safest and most practical bicycle facilities to facilitate cycle travel between Logan Road and the V1 south of Birdwood Road. e. Improve the awareness of the V1 Stage C through signage for cyclists approaching from the north with the aim of providing a better understanding of the route of the V1 to the south. f. Refine the use of motion activated traffic cameras to improve the capture rate of useable images and obtain an ongoing collection over time of V1 usage data. g. Undertake discussions with Strava, Inc. to refine the presentation of Strava data to improve visual understanding of maps showing before and after cycle route volumes along and on roads leading to the V1.

International trends in the implementation of assessment for learning: Implications for policy and practice

This paper discusses the emergence of assessment for learning (AfL) across the globe with particular attention given to Western educational jurisdictions. Authors from Australia, Canada, Ireland, Israel, New Zealand, Norway, and the USA explain the genesis of AfL, its evolution and impact on school systems, and discuss current trends in policy directions for AfL within their respective countries. The authors also discuss the implications of these various shifts and the ongoing tensions that exist between AfL and summative forms of assessment within national policy initiatives.

Design and evaluation of decision and control strategies for autonomous vision-based see and avoid systems

This paper details the design and performance assessment of a unique collision avoidance decision and control strategy for autonomous vision-based See and Avoid systems. The general approach revolves around re-positioning a collision object in the image using image-based visual servoing, without estimating range or time to collision. The decision strategy thus involves determining where to move the collision object, to induce a safe avoidance manuever, and when to cease the avoidance behaviour. These tasks are accomplished by exploiting human navigation models, spiral motion properties, expected image feature uncertainty and the rules of the air. The result is a simple threshold based system that can be tuned and statistically evaluated by extending performance assessment techniques derived for alerting systems. Our results demonstrate how autonomous vision-only See and Avoid systems may be designed under realistic problem constraints, and then evaluated in a manner consistent to aviation expectations.

Context-based assessment: Creating opportunities for resonance between classroom fields and societal fields

There is on-going international interest in the relationships between assessment instruments, students’ understanding of science concepts and context-based curriculum approaches. This study extends earlier research showing that students can develop connections between contexts and concepts – called fluid transitions – when studying context-based courses. We provide an in-depth investigation of one student’s experiences with multiple contextual assessment instruments that were associated with a context-based course. We analyzed the student’s responses to context-based assessment instruments to determine the extent to which contextual tests, reports of field investigations, and extended experimental investigations afforded her opportunities to make connections between contexts and concepts. A system of categorizing student responses was developed that can inform other educators when analyzing student responses to contextual assessment. We also refine the theoretical construct of fluid transitions that informed the study initially. Implications for curriculum and assessment design are provided in light of the findings.

In vitro/In vivo assessment and mechanisms of toxicity of bioceramic materials and its wear particulates

With the progress in modern technological research, novel biomaterials are being largely developed for various biomedical applications. Over the past two decades, most of the research focuses on the development of a new generation of bioceramics as substitutes for hard tissue replacement. In reference to their application in different anatomical locations of a patient, newly developed bioceramic materials can potentially induce a toxic/harmful effect to the host tissues. Therefore, prior to clinical testing, relevant biochemical screening assays are to be performed at the cellular and molecular level, to address the issues of biocompatibility and long term performance of the implants. Along with testing strategies in the bulk material toxicity, a detailed evaluation should also be conducted to determine the toxicity of the wear products of the potential bioceramics. This is important as the bioceramics are intended to be implanted in patients with longer life expectancy and notwithstanding, the material will eventually release finer (mostly nanosized) sized debris particles due to continuous wear at articulating surfaces in the hostile corrosive environment of the human body. The wear particulates generated from a biocompatible bioceramic may act in a different way, inducing early/late aseptic loosening at the implant site, resulting in osteolysis and inflammation. Hence, a study on the chronic effects of the wear particulates, in terms of local and systemic toxicity becomes the major criteria in the toxicity evaluation of implantable bioceramics. In this broad perspective, this article summarizes some of the currently used techniques and knowledge in assessing the in vitro and in vivo cytotoxicity and genotoxicity of bioceramic implant materials. It also addresses the need to conduct a broad evaluation before claiming the biocompatibility and clinical feasibility of any new biomaterial. This review also emphasizes some of the case studies based on the experimental designs that are currently followed and its importance in the context of clinical applications.

Fish Habitat Utilization Patterns and Evaluation of the Efficacy of Marine Protected Areas in Hawaii: Integration of NOAA Digital Benthic Habitat Mapping and Coral Reef Ecological Studies

Over the past four decades, the state of Hawaii has developed a system of eleven Marine Life Conservation Districts (MLCDs) to conserve and replenish marine resources around the state. Initially established to provide opportunities for public interaction with the marine environment, these MLCDs vary in size, habitat quality, and management regimes, providing an excellent opportunity to test hypotheses concerning marine protected area (MPA) design and function using multiple discreet sampling units. NOAA/NOS/NCCOS/Center for Coastal Monitoring and Assessment’s Biogeography Team developed digital benthic habitat maps for all MLCD and adjacent habitats. These maps were used to evaluate the efficacy of existing MLCDs for biodiversity conservation and fisheries replenishment, using a spatially explicit stratified random sampling design. Coupling the distribution of habitats and species habitat affinities using GIS technology elucidates species habitat utilization patterns at scales that are commensurate with ecosystem processes and is useful in defining essential fish habitat and biologically relevant boundaries for MPAs. Analysis of benthic cover validated the a priori classification of habitat types and provided justification for using these habitat strata to conduct stratified random sampling and analyses of fish habitat utilization patterns. Results showed that the abundance and distribution of species and assemblages exhibited strong correlations with habitat types. Fish assemblages in the colonized and uncolonized hardbottom habitats were found to be most similar among all of the habitat types. Much of the macroalgae habitat sampled was macroalgae growing on hard substrate, and as a result showed similarities with the other hardbottom assemblages. The fish assemblages in the sand habitats were highly variable but distinct from the other habitat types. Management regime also played an important role in the abundance and distribution of fish assemblages. MLCDs had higher values for most fish assemblage characteristics (e.g. biomass, size, diversity) compared with adjacent fished areas and Fisheries Management Areas (FMAs) across all habitat types. In addition, apex predators and other targeted resources species were more abundant and larger in the MLCDs, illustrating the effectiveness of these closures in conserving fish populations. Habitat complexity, quality, size and level of protection from fishing were important determinates of MLCD effectiveness with respect to their associated fish assemblages. (PDF contains 217 pages)

Monitoring and evaluation of management effectiveness for adaptive management of MPAs in Indonesia's BOBLME area

Subcomponent 3.2 Marine Protected Areas in the conservation of regional fish stocks "Provision of Services Relating to the Assessment and Planning of Effective Management of two MPAs in Sumatra Coast, Indonesia". Assessments of management effectiveness of two Marine Protected Areas (MPAs) in Pulau Pieh National Park and Sabang District, Sumatra, were conducted through two separate workshops. Recommendations were made and future needs identified.