Reframing evaluation of learning and teaching: An approach to change

Evaluation practices in the Higher Education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology (QUT) began a project which aims to reframe the organisation’s approach to the evaluation of learning and teaching. This paper describes the existing evaluation system; the emergence and early development of the project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature, to determine its validity and suitability for supporting QUT’s plans for the future. Overall, the paper represents a structured evaluation of the REFRAME project at a particular point in its lifecycle. Given that the project follows an evidence based, practice-led process and applies an ongoing action research cycle, the findings are presented in the belief that QUT’s experience is broadly applicable to other institutions which may be contemplating change in relation to evaluation of learning and teaching.

Evaluation of screening tests for predicting older driver performance and safety assessed by an on-road test

A number of tests and test batteries are available for the prediction of older driver safety, but many of these have not been validated against standardized driving outcome measures. The aim of this study was to evaluate a series of previously described screening tests in terms of their ability to predict the potential for safe and unsafe driving. Participants included 79 community-dwelling older drivers (M=72.16 years, SD=5.46; range 65-88 years; 57 males and 22 females) who completed a previously validated multi-disciplinary driving assessment, a hazard perception test, a hazard change detection test and a battery of vision and cognitive tests. Participants also completed a standardized on-road driving assessment. The multi-disciplinary test battery had the highest predictive ability with a sensitivity of 80% and a specificity of 73%, followed by the hazard perception test which demonstrated a sensitivity of 75% and a specificity of 61%. These findings suggest that a relatively simple and practical battery of tests from a range of domains has the capacity to predict safe and unsafe driving in older adults.

Evaluation of the cost-effectiveness of strategies claiming to reduce the risk of surgical site infections following primary total hip arthroplasty

Background Total hip arthroplasty (THA) is a commonly performed procedure and numbers are increasing with ageing populations. One of the most serious complications in THA are surgical site infections (SSIs), caused by pathogens entering the wound during the procedure. SSIs are associated with a substantial burden for health services, increased mortality and reduced functional outcomes in patients. Numerous approaches to preventing these infections exist but there is no gold standard in practice and the cost-effectiveness of alternate strategies is largely unknown. Objectives The aim of this project was to evaluate the cost-effectiveness of strategies claiming to reduce deep surgical site infections following total hip arthroplasty in Australia. The objectives were: 1. Identification of competing strategies or combinations of strategies that are clinically relevant to the control of SSI related to hip arthroplasty 2. Evidence synthesis and pooling of results to assess the volume and quality of evidence claiming to reduce the risk of SSI following total hip arthroplasty 3. Construction of an economic decision model incorporating cost and health outcomes for each of the identified strategies 4. Quantification of the effect of uncertainty in the model 5. Assessment of the value of perfect information among model parameters to inform future data collection Methods The literature relating to SSI in THA was reviewed, in particular to establish definitions of these concepts, understand mechanisms of aetiology and microbiology, risk factors, diagnosis and consequences as well as to give an overview of existing infection prevention measures. Published economic evaluations on this topic were also reviewed and limitations for Australian decision-makers identified. A Markov state-transition model was developed for the Australian context and subsequently validated by clinicians. The model was designed to capture key events related to deep SSI occurring within the first 12 months following primary THA. Relevant infection prevention measures were selected by reviewing clinical guideline recommendations combined with expert elicitation. Strategies selected for evaluation were the routine use of pre-operative antibiotic prophylaxis (AP) versus no use of antibiotic prophylaxis (No AP) or in combination with antibiotic-impregnated cement (AP & ABC) or laminar air operating rooms (AP & LOR). The best available evidence for clinical effect size and utility parameters was harvested from the medical literature using reproducible methods. Queensland hospital data were extracted to inform patients’ transitions between model health states and related costs captured in assigned treatment codes. Costs related to infection prevention were derived from reliable hospital records and expert opinion. Uncertainty of model input parameters was explored in probabilistic sensitivity analyses and scenario analyses and the value of perfect information was estimated. Results The cost-effectiveness analysis was performed from a health services perspective using a hypothetical cohort of 30,000 THA patients aged 65 years. The baseline rate of deep SSI was 0.96% within one year of a primary THA. The routine use of antibiotic prophylaxis (AP) was highly cost-effective and resulted in cost savings of over $1.6m whilst generating an extra 163 QALYs (without consideration of uncertainty). Deterministic and probabilistic analysis (considering uncertainty) identified antibiotic prophylaxis combined with antibiotic-impregnated cement (AP & ABC) to be the most cost-effective strategy. Using AP & ABC generated the highest net monetary benefit (NMB) and an incremental $3.1m NMB compared to only using antibiotic prophylaxis. There was a very low error probability that this strategy might not have the largest NMB (<5%). Not using antibiotic prophylaxis (No AP) or using both antibiotic prophylaxis combined with laminar air operating rooms (AP & LOR) resulted in worse health outcomes and higher costs. Sensitivity analyses showed that the model was sensitive to the initial cohort starting age and the additional costs of ABC but the best strategy did not change, even for extreme values. The cost-effectiveness improved for a higher proportion of cemented primary THAs and higher baseline rates of deep SSI. The value of perfect information indicated that no additional research is required to support the model conclusions. Conclusions Preventing deep SSI with antibiotic prophylaxis and antibiotic-impregnated cement has shown to improve health outcomes among hospitalised patients, save lives and enhance resource allocation. By implementing a more beneficial infection control strategy, scarce health care resources can be used more efficiently to the benefit of all members of society. The results of this project provide Australian policy makers with key information about how to efficiently manage risks of infection in THA.

Looking Yonder : Longitudinal Research and Evaluation of the Yonder Participative Arts-based Learning Model Report 2010-2012

Since 2007, KITE Arts Education Program @ QPAC has been engaged in a series of arts and drama-based experiences for students in selected primary schools on the edges of Brisbane and in regional Queensland. The in-school workshop experiences of the program have culminated in a performance by the children for their school community, parents and carers at the Queensland Performing Arts Centre or a regional cultural venue. In conducting an analysis of the Yonder project, the researcher aimed to provide evidence of outcomes brought about through participation by schools, school staff, students and their communities in the Yonder project. To develop longitudinal data project initiators, participants were interviewed at six-monthly intervals to establish patterns of engagement and participation. The report analyses arts-based workshops conducted by the teacher artist in edge-city Brisbane and a regional centre; interviews with teachers and school administrators from the participating schools; interviews with teacher artist and professional artists; interviews with community partners; teacher professional development workshops; community-based workshops; performance outcomes that were the culminating events of the workshop program; student work samples and student reflections on the program. This document covers data and project outputs from February 2010 to July 2012. There have been five iterations of the Yonder project since its commencement in mid-2009 — three in regional Queensland (February–April 2010; February–May 2011; February–May 2012) and two in edge-city1 Brisbane (July–September 2010; August–October 2011). This report is a result of a research partnership between Queensland Performing Arts Centre and Queensland University of Technology (QUT) Creative Industries Faculty(Drama).

Nifty after fifty: evaluation of a physical activity directory for older people

Objective: To develop a physical activity directory (PAD) for Brisbane people over the age of 50 years for distribution by two methods (given or requested), and to determine its effectiveness in raising awareness and encouraging older people to participate in local physical activity options. Methods: Baseline demographic data and stage of change was collected from 224 participants who received the directory. Participants were interviewed by telephone six weeks later to determine their use of the directory on a number of dimensions. Results: Most participants interviewed at follow-up remembered reading the directory. Participants who requested the directory were significantly more likely than those who were given it to: be contemplators, read the directory, plan to ring a number, plan to attend a class, and to share the directory with others. Participants who were contemplators were significantly more likely to have participated in physical activity of their own and rang a number from the directory. The directory increased over half the participants' awareness of local physical activity options, yet only 7% reported ringing a number and 15% reported doing their own physical activity. Conclusions: The directory was more effective in raising awareness about physical activity options than encouraging people to participate in physical activity, and participants with short-term plans to be more active were more likely to have used the directory. Implications: The directory, even when linked with other services, raises awareness about physical activity options, but has minimal short-term influence on participation.

Reaching high-risk adolescents : a process evaluation of a school based injury prevention program

High-risk adolescents are a population most vulnerable to harm from injury due to increased engagement in risk taking behaviour. There is a gap in the literature regarding how universal school based injury prevention programs apply to high-risk adolescents. This study involves a component of the process evaluation of a school based injury prevention program, as it relates to high-risk adolescents (13-14 years)...

Community development with survivors of torture and trauma : an evaluation framework

Despite an increase in community development initiatives in refugee contexts, there is a lack of evaluation frameworks to assess the effectiveness of interventions in the recovery of refugee communities. In response to this gap, the Forum of Australian Services for Survivors of Torture and Trauma has developed an evaluation framework in consultation with refugee client groups and agencies' staff members. This paper contextualizes the goals, principles and strategies of services implementing community development initiatives with torture and trauma survivors and describes the process of developing the framework within a participatory action approach. Both outcome evaluation and process evaluation are discussed, and examples of the framework are presented. Community development agencies and professionals working with survivors of torture and trauma can play a significant role by fostering community empowerment through evaluation.

Evaluation of implementation of a healthy food and drink supply strategy throughout the whole school environment in Queensland state schools, Australia

BACKGROUND/OBJECTIVES: This paper reports on the evaluation of the Smart Choices healthy food and drink supply strategy for Queensland schools (Smart Choices) implementation across the whole school environment in state government primary and secondary schools in Queensland, Australia. SUBJECTS/METHODS: Three concurrent surveys using different methods for each group of stakeholders that targeted all 1275 school Principals, all 1258 Parent and Citizens’ Associations (P&Cs) and a random sample of 526 tuckshop convenors throughout Queensland. Nine hundred and seventy-three Principals, 598 P&Cs and 513 tuckshop convenors participated with response rates of 78%, 48% and 98%, respectively. RESULTS: Nearly all Principals (97%), P&Cs (99%) and tuckshop convenors (97%) reported that their school tuckshop had implemented Smart Choices. The majority of Principals and P&Cs reported implementation, respectively, in: school breakfast programs (98 and 92%); vending machine stock (94 and 83%); vending machine advertising (85 and 84%); school events (87 and 88%); school sporting events (81 and 80%); sponsorship and advertising (93 and 84%); fundraising events (80 and 84%); and sporting clubs (73 and 75%). Implementation in curriculum activities, classroom rewards and class parties was reported, respectively, by 97%, 86% and 75% of Principals. Respondents also reported very high levels of understanding of Smart Choices and engagement of the school community. CONCLUSIONS: The results demonstrated that food supply interventions to promote nutrition across all domains of the school environment can be implemented successfully.

Evaluation of new and existing confidence measures for stereo matching

This paper outlines existing matching diagnostics, which may be used for identifying invalid matches and estimating the probability of a correct match. In addition, it proposes a new diagnostic for error prediction which can be used with the rank and census transforms. Both the existing and the new diagnostics have been evaluated and compared for a number of test images. In each case, a confidence estimate was computed for every location of the disparity map, and disparities having a low confidence estimate removed from the disparity map. Collectively, these confidence estimates may be termed a confidence map. Such information would be useful for potential applications of stereo vision such as automation and navigation.

Quantitative evaluation of matching methods and validity measures for stereo vision

The authors present a qualitative and quantitative comparison of various similarity measures that form the kernel of common area-based stereo-matching systems. The authors compare classical difference and correlation measures as well as nonparametric measures based on the rank and census transforms for a number of outdoor images. For robotic applications, important considerations include robustness to image defects such as intensity variation and noise, the number of false matches, and computational complexity. In the absence of ground truth data, the authors compare the matching techniques based on the percentage of matches that pass the left-right consistency test. The authors also evaluate the discriminatory power of several match validity measures that are reported in the literature for eliminating false matches and for estimating match confidence. For guidance applications, it is essential to have and estimate of confidence in the three-dimensional points generated by stereo vision. Finally, a new validity measure, the rank constraint, is introduced that is capable of resolving ambiguous matches for rank transform-based matching.

The effect of alcohol and drug consumption on academic performance : a treatment effect evaluation

It is often argued that consumption of alcohol, tobacco and drugs is detrimental to the cognitive abilities of teenagers. In order to disentangle a possible causal effect of these substances use from a self-selection bias, we control for pupils previous performance and for their previous rate of progression applying a DiDiD strategy. Using the NELS 1988 panel dataset, we find that the effects of alcohol and tobacco on test scores disappear once the selection bias is controlled for (this does not preclude long term detrimental effects). However, we find reliable evidence that heavy use of drugs (marijuana and cocaine) has direct detrimental effects on educational achievements. Hence, our results may have significant policy implications.

Evaluation of "Eat Well Queensland"

Eat Well Queensland 2002-2012: Smart Eating for a Healthier State (EWQ) was developed by the Queensland Public Health Forum in 2002 as a 10-year strategy to improve the health of Queenslanders through better food and nutrition. This study aimed to evaluate the implementation of EWQ and identify future strategic action required. Queensland Health funded a mid-point review of EWQ in 2008, to identify achievements, gaps, barriers and emerging issues associated with EWQ. 31 key stakeholders were interviewed, 83 stakeholders responded to an online survey, 150 stakeholders attended a state-wide practitioner workshop and 209 EWQ-related project reports were assessed.