Reframing evaluation of learning and teaching: An approach to change

Evaluation practices in the Higher Education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology (QUT) began a project which aims to reframe the organisation’s approach to the evaluation of learning and teaching. This paper describes the existing evaluation system; the emergence and early development of the project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature, to determine its validity and suitability for supporting QUT’s plans for the future. Overall, the paper represents a structured evaluation of the REFRAME project at a particular point in its lifecycle. Given that the project follows an evidence based, practice-led process and applies an ongoing action research cycle, the findings are presented in the belief that QUT’s experience is broadly applicable to other institutions which may be contemplating change in relation to evaluation of learning and teaching.

Evaluation of screening tests for predicting older driver performance and safety assessed by an on-road test

A number of tests and test batteries are available for the prediction of older driver safety, but many of these have not been validated against standardized driving outcome measures. The aim of this study was to evaluate a series of previously described screening tests in terms of their ability to predict the potential for safe and unsafe driving. Participants included 79 community-dwelling older drivers (M=72.16 years, SD=5.46; range 65-88 years; 57 males and 22 females) who completed a previously validated multi-disciplinary driving assessment, a hazard perception test, a hazard change detection test and a battery of vision and cognitive tests. Participants also completed a standardized on-road driving assessment. The multi-disciplinary test battery had the highest predictive ability with a sensitivity of 80% and a specificity of 73%, followed by the hazard perception test which demonstrated a sensitivity of 75% and a specificity of 61%. These findings suggest that a relatively simple and practical battery of tests from a range of domains has the capacity to predict safe and unsafe driving in older adults.

Evaluation of the cost-effectiveness of strategies claiming to reduce the risk of surgical site infections following primary total hip arthroplasty

Background Total hip arthroplasty (THA) is a commonly performed procedure and numbers are increasing with ageing populations. One of the most serious complications in THA are surgical site infections (SSIs), caused by pathogens entering the wound during the procedure. SSIs are associated with a substantial burden for health services, increased mortality and reduced functional outcomes in patients. Numerous approaches to preventing these infections exist but there is no gold standard in practice and the cost-effectiveness of alternate strategies is largely unknown. Objectives The aim of this project was to evaluate the cost-effectiveness of strategies claiming to reduce deep surgical site infections following total hip arthroplasty in Australia. The objectives were: 1. Identification of competing strategies or combinations of strategies that are clinically relevant to the control of SSI related to hip arthroplasty 2. Evidence synthesis and pooling of results to assess the volume and quality of evidence claiming to reduce the risk of SSI following total hip arthroplasty 3. Construction of an economic decision model incorporating cost and health outcomes for each of the identified strategies 4. Quantification of the effect of uncertainty in the model 5. Assessment of the value of perfect information among model parameters to inform future data collection Methods The literature relating to SSI in THA was reviewed, in particular to establish definitions of these concepts, understand mechanisms of aetiology and microbiology, risk factors, diagnosis and consequences as well as to give an overview of existing infection prevention measures. Published economic evaluations on this topic were also reviewed and limitations for Australian decision-makers identified. A Markov state-transition model was developed for the Australian context and subsequently validated by clinicians. The model was designed to capture key events related to deep SSI occurring within the first 12 months following primary THA. Relevant infection prevention measures were selected by reviewing clinical guideline recommendations combined with expert elicitation. Strategies selected for evaluation were the routine use of pre-operative antibiotic prophylaxis (AP) versus no use of antibiotic prophylaxis (No AP) or in combination with antibiotic-impregnated cement (AP & ABC) or laminar air operating rooms (AP & LOR). The best available evidence for clinical effect size and utility parameters was harvested from the medical literature using reproducible methods. Queensland hospital data were extracted to inform patients’ transitions between model health states and related costs captured in assigned treatment codes. Costs related to infection prevention were derived from reliable hospital records and expert opinion. Uncertainty of model input parameters was explored in probabilistic sensitivity analyses and scenario analyses and the value of perfect information was estimated. Results The cost-effectiveness analysis was performed from a health services perspective using a hypothetical cohort of 30,000 THA patients aged 65 years. The baseline rate of deep SSI was 0.96% within one year of a primary THA. The routine use of antibiotic prophylaxis (AP) was highly cost-effective and resulted in cost savings of over $1.6m whilst generating an extra 163 QALYs (without consideration of uncertainty). Deterministic and probabilistic analysis (considering uncertainty) identified antibiotic prophylaxis combined with antibiotic-impregnated cement (AP & ABC) to be the most cost-effective strategy. Using AP & ABC generated the highest net monetary benefit (NMB) and an incremental $3.1m NMB compared to only using antibiotic prophylaxis. There was a very low error probability that this strategy might not have the largest NMB (<5%). Not using antibiotic prophylaxis (No AP) or using both antibiotic prophylaxis combined with laminar air operating rooms (AP & LOR) resulted in worse health outcomes and higher costs. Sensitivity analyses showed that the model was sensitive to the initial cohort starting age and the additional costs of ABC but the best strategy did not change, even for extreme values. The cost-effectiveness improved for a higher proportion of cemented primary THAs and higher baseline rates of deep SSI. The value of perfect information indicated that no additional research is required to support the model conclusions. Conclusions Preventing deep SSI with antibiotic prophylaxis and antibiotic-impregnated cement has shown to improve health outcomes among hospitalised patients, save lives and enhance resource allocation. By implementing a more beneficial infection control strategy, scarce health care resources can be used more efficiently to the benefit of all members of society. The results of this project provide Australian policy makers with key information about how to efficiently manage risks of infection in THA.

Looking Yonder : Longitudinal Research and Evaluation of the Yonder Participative Arts-based Learning Model Report 2010-2012

Since 2007, KITE Arts Education Program @ QPAC has been engaged in a series of arts and drama-based experiences for students in selected primary schools on the edges of Brisbane and in regional Queensland. The in-school workshop experiences of the program have culminated in a performance by the children for their school community, parents and carers at the Queensland Performing Arts Centre or a regional cultural venue. In conducting an analysis of the Yonder project, the researcher aimed to provide evidence of outcomes brought about through participation by schools, school staff, students and their communities in the Yonder project. To develop longitudinal data project initiators, participants were interviewed at six-monthly intervals to establish patterns of engagement and participation. The report analyses arts-based workshops conducted by the teacher artist in edge-city Brisbane and a regional centre; interviews with teachers and school administrators from the participating schools; interviews with teacher artist and professional artists; interviews with community partners; teacher professional development workshops; community-based workshops; performance outcomes that were the culminating events of the workshop program; student work samples and student reflections on the program. This document covers data and project outputs from February 2010 to July 2012. There have been five iterations of the Yonder project since its commencement in mid-2009 — three in regional Queensland (February–April 2010; February–May 2011; February–May 2012) and two in edge-city1 Brisbane (July–September 2010; August–October 2011). This report is a result of a research partnership between Queensland Performing Arts Centre and Queensland University of Technology (QUT) Creative Industries Faculty(Drama).

Viable chimaeric viruses confirm the biological importance of sequence specific maize streak virus movement protein and coat protein interactions

Background. A variety of interactions between up to three different movement proteins (MPs), the coat protein (CP) and genomic DNA mediate the inter- and intra-cellular movement of geminiviruses in the genus Begomovirus. Although movement of viruses in the genus Mastrevirus is less well characterized, direct interactions between a single MP and the CP of these viruses is also clearly involved in both intra- and intercellular trafficking of virus genomic DNA. However, it is currently unknown how specific these MP-CP interactions are, nor how disruption of these interactions might impact on virus viability. Results. Using chimaeric genomes of two strains of Maize streak virus (MSV) we adopted a genetic approach to investigate the gross biological effects of interfering with interactions between virus MP and CP homologues derived from genetically distinct MSV isolates. MP and CP genes were reciprocally exchanged, individually and in pairs, between maize (MSV-Kom)- and Setaria sp. (MSV-Set)-adapted isolates sharing 78% genome-wide sequence identity. All chimaeras were infectious in Zea mays c.v. Jubilee and were characterized in terms of symptomatology and infection efficiency. Compared with their parental viruses, all the chimaeras were attenuated in symptom severity, infection efficiency, and the rate at which symptoms appeared. The exchange of individual MP and CP genes resulted in lower infection efficiency and reduced symptom severity in comparison with exchanges of matched MP-CP pairs. Conclusion. Specific interactions between the mastrevirus MP and CP genes themselves and/or their expression products are important determinants of infection efficiency, rate of symptom development and symptom severity. © 2008 van der Walt et al; licensee BioMed Central Ltd.

Nifty after fifty: evaluation of a physical activity directory for older people

Objective: To develop a physical activity directory (PAD) for Brisbane people over the age of 50 years for distribution by two methods (given or requested), and to determine its effectiveness in raising awareness and encouraging older people to participate in local physical activity options. Methods: Baseline demographic data and stage of change was collected from 224 participants who received the directory. Participants were interviewed by telephone six weeks later to determine their use of the directory on a number of dimensions. Results: Most participants interviewed at follow-up remembered reading the directory. Participants who requested the directory were significantly more likely than those who were given it to: be contemplators, read the directory, plan to ring a number, plan to attend a class, and to share the directory with others. Participants who were contemplators were significantly more likely to have participated in physical activity of their own and rang a number from the directory. The directory increased over half the participants' awareness of local physical activity options, yet only 7% reported ringing a number and 15% reported doing their own physical activity. Conclusions: The directory was more effective in raising awareness about physical activity options than encouraging people to participate in physical activity, and participants with short-term plans to be more active were more likely to have used the directory. Implications: The directory, even when linked with other services, raises awareness about physical activity options, but has minimal short-term influence on participation.

Reaching high-risk adolescents : a process evaluation of a school based injury prevention program

High-risk adolescents are a population most vulnerable to harm from injury due to increased engagement in risk taking behaviour. There is a gap in the literature regarding how universal school based injury prevention programs apply to high-risk adolescents. This study involves a component of the process evaluation of a school based injury prevention program, as it relates to high-risk adolescents (13-14 years)...

Community development with survivors of torture and trauma : an evaluation framework

Despite an increase in community development initiatives in refugee contexts, there is a lack of evaluation frameworks to assess the effectiveness of interventions in the recovery of refugee communities. In response to this gap, the Forum of Australian Services for Survivors of Torture and Trauma has developed an evaluation framework in consultation with refugee client groups and agencies' staff members. This paper contextualizes the goals, principles and strategies of services implementing community development initiatives with torture and trauma survivors and describes the process of developing the framework within a participatory action approach. Both outcome evaluation and process evaluation are discussed, and examples of the framework are presented. Community development agencies and professionals working with survivors of torture and trauma can play a significant role by fostering community empowerment through evaluation.

Evaluation of implementation of a healthy food and drink supply strategy throughout the whole school environment in Queensland state schools, Australia

BACKGROUND/OBJECTIVES: This paper reports on the evaluation of the Smart Choices healthy food and drink supply strategy for Queensland schools (Smart Choices) implementation across the whole school environment in state government primary and secondary schools in Queensland, Australia. SUBJECTS/METHODS: Three concurrent surveys using different methods for each group of stakeholders that targeted all 1275 school Principals, all 1258 Parent and Citizens’ Associations (P&Cs) and a random sample of 526 tuckshop convenors throughout Queensland. Nine hundred and seventy-three Principals, 598 P&Cs and 513 tuckshop convenors participated with response rates of 78%, 48% and 98%, respectively. RESULTS: Nearly all Principals (97%), P&Cs (99%) and tuckshop convenors (97%) reported that their school tuckshop had implemented Smart Choices. The majority of Principals and P&Cs reported implementation, respectively, in: school breakfast programs (98 and 92%); vending machine stock (94 and 83%); vending machine advertising (85 and 84%); school events (87 and 88%); school sporting events (81 and 80%); sponsorship and advertising (93 and 84%); fundraising events (80 and 84%); and sporting clubs (73 and 75%). Implementation in curriculum activities, classroom rewards and class parties was reported, respectively, by 97%, 86% and 75% of Principals. Respondents also reported very high levels of understanding of Smart Choices and engagement of the school community. CONCLUSIONS: The results demonstrated that food supply interventions to promote nutrition across all domains of the school environment can be implemented successfully.

A micro-level indexing model for assessing urban ecosystem sustainability

Purpose – As a consequence of rapid urbanisation and globalisation, cities have become the engines of population and economic growth. Hence, natural resources in and around the cities have been exposed to externalities of urban development processes. This paper introduces a new sustainability assessment approach that is tested in a pilot study. The paper aims to assist policy-makers and planners investigating the impacts of development on environmental systems, and produce effective policies for sustainable urban development. Design/methodology/approach – The paper introduces an indicator-based indexing model entitled “Indexing Model for the Assessment of Sustainable Urban Ecosystems” (ASSURE). The ASSURE indexing model produces a set of micro-level environmental sustainability indices that is aimed to be used in the evaluation and monitoring of the interaction between human activities and urban ecosystems. The model is an innovative approach designed to assess the resilience of ecosystems towards impacts of current development plans and the results serve as a guide for policymakers to take actions towards achieving sustainability. Findings – The indexing model has been tested in a pilot case study within the Gold Coast City, Queensland, Australia. This paper presents the methodology of the model and outlines the preliminary findings of the pilot study. The paper concludes with a discussion on the findings and recommendations put forward for future development and implementation of the model. Originality/value – Presently, there is a few sustainability indices developed to measure the sustainability at local, regional, national and international levels. However, due to challenges in data collection difficulties and availability of local data, there is no effective assessment model at the microlevel that the assessment of urban ecosystem sustainability accurately. The model introduced in this paper fills this gap by focusing on parcel-scale and benchmarking the environmental performance in micro-level.

WSUD application audit at Gold Coast City

In recent years, Water Sensitive Urban Design (WSUD) has been strongly promoted in South East Queensland to mitigate quantity and quality issues in relation to stormwater. Gold Coast City Council has implemented WSUD devices widely for stormwater management for a number of years and is planning to continue this practice into the future. According to the planning policy of Gold Coast City Council, the adoption of WSUD practices is now mandatory for any new development within the city. As a result, Council is expected to be in possession of tens of millions of dollars of these assets in the future and will be responsible for their maintenance and long-term management. Any shortcoming in the implementation of best practice can potentially result in substantial liability for the Council in the future. However, there has been limited evaluation of WSUD systems in relation to their performance, long-term maintenance, and current knowledge gaps. It was considered that periodical audits of WSUD applications on the Gold Coast is vital to ensure that Council’s WSUD policies are continually improved to new learning and best practice is implemented and risk to Council is mitigated. After a series of stakeholder interviews within Council to understand current practical issues (weaknesses and strengths) in relation to the implementation of WSUD on the Gold Coast, a field audit comprising of condition assessment of eleven WSUD systems within four suburbs was undertaken to identify weaknesses and strengths in WSUD implementation on the Gold Coast. The outcomes of this study are presented in this paper.

Automatic particle picking of biological molecules imaged by electron microscopy

One of the next great challenges of cell biology is the determination of the enormous number of protein structures encoded in genomes. In recent years, advances in electron cryo-microscopy and high-resolution single particle analysis have developed to the point where they now provide a methodology for high resolution structure determination. Using this approach, images of randomly oriented single particles are aligned computationally to reconstruct 3-D structures of proteins and even whole viruses. One of the limiting factors in obtaining high-resolution reconstructions is obtaining a large enough representative dataset ($>100,000$ particles). Traditionally particles have been manually picked which is an extremely labour intensive process. The problem is made especially difficult by the low signal-to-noise ratio of the images. This paper describes the development of automatic particle picking software, which has been tested with both negatively stained and cryo-electron micrographs. This algorithm has been shown to be capable of selecting most of the particles, with few false positives. Further work will involve extending the software to detect differently shaped and oriented particles.

Evaluation of new and existing confidence measures for stereo matching

This paper outlines existing matching diagnostics, which may be used for identifying invalid matches and estimating the probability of a correct match. In addition, it proposes a new diagnostic for error prediction which can be used with the rank and census transforms. Both the existing and the new diagnostics have been evaluated and compared for a number of test images. In each case, a confidence estimate was computed for every location of the disparity map, and disparities having a low confidence estimate removed from the disparity map. Collectively, these confidence estimates may be termed a confidence map. Such information would be useful for potential applications of stereo vision such as automation and navigation.