Evaluation of the quality of diverse teaching practice in higher education research methods

The purpose of this paper is to report on a methods research project investigating the evaluation of diverse teaching practice in Higher Education. The research method is a single site case study of an Australian university with data collected through published documents, surveys, interviews and focus groups. This project provides evidence of the wide variety of evaluation practice and diverse teaching practice across the university. This breadth identifies the need for greater flexibility of evaluation processes, tools and support to assist teaching staff to evaluate their diverse teaching practice. The employment opportunities for academics benchmark the university nationally and position the case study in the field. Finally this reaffirms the institutional responsibility for services to support teaching staff in an ongoing manner.

Reframing evaluation of learning and teaching: An approach to change

Evaluation practices in the Higher Education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology (QUT) began a project which aims to reframe the organisation’s approach to the evaluation of learning and teaching. This paper describes the existing evaluation system; the emergence and early development of the project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature, to determine its validity and suitability for supporting QUT’s plans for the future. Overall, the paper represents a structured evaluation of the REFRAME project at a particular point in its lifecycle. Given that the project follows an evidence based, practice-led process and applies an ongoing action research cycle, the findings are presented in the belief that QUT’s experience is broadly applicable to other institutions which may be contemplating change in relation to evaluation of learning and teaching.

Evaluation of screening tests for predicting older driver performance and safety assessed by an on-road test

A number of tests and test batteries are available for the prediction of older driver safety, but many of these have not been validated against standardized driving outcome measures. The aim of this study was to evaluate a series of previously described screening tests in terms of their ability to predict the potential for safe and unsafe driving. Participants included 79 community-dwelling older drivers (M=72.16 years, SD=5.46; range 65-88 years; 57 males and 22 females) who completed a previously validated multi-disciplinary driving assessment, a hazard perception test, a hazard change detection test and a battery of vision and cognitive tests. Participants also completed a standardized on-road driving assessment. The multi-disciplinary test battery had the highest predictive ability with a sensitivity of 80% and a specificity of 73%, followed by the hazard perception test which demonstrated a sensitivity of 75% and a specificity of 61%. These findings suggest that a relatively simple and practical battery of tests from a range of domains has the capacity to predict safe and unsafe driving in older adults.

Evaluation of the cost-effectiveness of strategies claiming to reduce the risk of surgical site infections following primary total hip arthroplasty

Background Total hip arthroplasty (THA) is a commonly performed procedure and numbers are increasing with ageing populations. One of the most serious complications in THA are surgical site infections (SSIs), caused by pathogens entering the wound during the procedure. SSIs are associated with a substantial burden for health services, increased mortality and reduced functional outcomes in patients. Numerous approaches to preventing these infections exist but there is no gold standard in practice and the cost-effectiveness of alternate strategies is largely unknown. Objectives The aim of this project was to evaluate the cost-effectiveness of strategies claiming to reduce deep surgical site infections following total hip arthroplasty in Australia. The objectives were: 1. Identification of competing strategies or combinations of strategies that are clinically relevant to the control of SSI related to hip arthroplasty 2. Evidence synthesis and pooling of results to assess the volume and quality of evidence claiming to reduce the risk of SSI following total hip arthroplasty 3. Construction of an economic decision model incorporating cost and health outcomes for each of the identified strategies 4. Quantification of the effect of uncertainty in the model 5. Assessment of the value of perfect information among model parameters to inform future data collection Methods The literature relating to SSI in THA was reviewed, in particular to establish definitions of these concepts, understand mechanisms of aetiology and microbiology, risk factors, diagnosis and consequences as well as to give an overview of existing infection prevention measures. Published economic evaluations on this topic were also reviewed and limitations for Australian decision-makers identified. A Markov state-transition model was developed for the Australian context and subsequently validated by clinicians. The model was designed to capture key events related to deep SSI occurring within the first 12 months following primary THA. Relevant infection prevention measures were selected by reviewing clinical guideline recommendations combined with expert elicitation. Strategies selected for evaluation were the routine use of pre-operative antibiotic prophylaxis (AP) versus no use of antibiotic prophylaxis (No AP) or in combination with antibiotic-impregnated cement (AP & ABC) or laminar air operating rooms (AP & LOR). The best available evidence for clinical effect size and utility parameters was harvested from the medical literature using reproducible methods. Queensland hospital data were extracted to inform patients’ transitions between model health states and related costs captured in assigned treatment codes. Costs related to infection prevention were derived from reliable hospital records and expert opinion. Uncertainty of model input parameters was explored in probabilistic sensitivity analyses and scenario analyses and the value of perfect information was estimated. Results The cost-effectiveness analysis was performed from a health services perspective using a hypothetical cohort of 30,000 THA patients aged 65 years. The baseline rate of deep SSI was 0.96% within one year of a primary THA. The routine use of antibiotic prophylaxis (AP) was highly cost-effective and resulted in cost savings of over $1.6m whilst generating an extra 163 QALYs (without consideration of uncertainty). Deterministic and probabilistic analysis (considering uncertainty) identified antibiotic prophylaxis combined with antibiotic-impregnated cement (AP & ABC) to be the most cost-effective strategy. Using AP & ABC generated the highest net monetary benefit (NMB) and an incremental $3.1m NMB compared to only using antibiotic prophylaxis. There was a very low error probability that this strategy might not have the largest NMB (<5%). Not using antibiotic prophylaxis (No AP) or using both antibiotic prophylaxis combined with laminar air operating rooms (AP & LOR) resulted in worse health outcomes and higher costs. Sensitivity analyses showed that the model was sensitive to the initial cohort starting age and the additional costs of ABC but the best strategy did not change, even for extreme values. The cost-effectiveness improved for a higher proportion of cemented primary THAs and higher baseline rates of deep SSI. The value of perfect information indicated that no additional research is required to support the model conclusions. Conclusions Preventing deep SSI with antibiotic prophylaxis and antibiotic-impregnated cement has shown to improve health outcomes among hospitalised patients, save lives and enhance resource allocation. By implementing a more beneficial infection control strategy, scarce health care resources can be used more efficiently to the benefit of all members of society. The results of this project provide Australian policy makers with key information about how to efficiently manage risks of infection in THA.

Spectrum sensing, detection and optimisation in cognitive radio for non-stationary primary user signals

Cognitive radio is an emerging technology proposing the concept of dynamic spec- trum access as a solution to the looming problem of spectrum scarcity caused by the growth in wireless communication systems. Under the proposed concept, non- licensed, secondary users (SU) can access spectrum owned by licensed, primary users (PU) so long as interference to PU are kept minimal. Spectrum sensing is a crucial task in cognitive radio whereby the SU senses the spectrum to detect the presence or absence of any PU signal. Conventional spectrum sensing assumes the PU signal as ‘stationary’ and remains in the same activity state during the sensing cycle, while an emerging trend models PU as ‘non-stationary’ and undergoes state changes. Existing studies have focused on non-stationary PU during the transmission period, however very little research considered the impact on spectrum sensing when the PU is non-stationary during the sensing period. The concept of PU duty cycle is developed as a tool to analyse the performance of spectrum sensing detectors when detecting non-stationary PU signals. New detectors are also proposed to optimise detection with respect to duty cycle ex- hibited by the PU. This research consists of two major investigations. The first stage investigates the impact of duty cycle on the performance of existing detec- tors and the extent of the problem in existing studies. The second stage develops new detection models and frameworks to ensure the integrity of spectrum sensing when detecting non-stationary PU signals. The first investigation demonstrates that conventional signal model formulated for stationary PU does not accurately reflect the behaviour of a non-stationary PU. Therefore the performance calculated and assumed to be achievable by the conventional detector does not reflect actual performance achieved. Through analysing the statistical properties of duty cycle, performance degradation is proved to be a problem that cannot be easily neglected in existing sensing studies when PU is modelled as non-stationary. The second investigation presents detectors that are aware of the duty cycle ex- hibited by a non-stationary PU. A two stage detection model is proposed to improve the detection performance and robustness to changes in duty cycle. This detector is most suitable for applications that require long sensing periods. A second detector, the duty cycle based energy detector is formulated by integrat- ing the distribution of duty cycle into the test statistic of the energy detector and suitable for short sensing periods. The decision threshold is optimised with respect to the traffic model of the PU, hence the proposed detector can calculate average detection performance that reflect realistic results. A detection framework for the application of spectrum sensing optimisation is proposed to provide clear guidance on the constraints on sensing and detection model. Following this framework will ensure the signal model accurately reflects practical behaviour while the detection model implemented is also suitable for the desired detection assumption. Based on this framework, a spectrum sensing optimisation algorithm is further developed to maximise the sensing efficiency for non-stationary PU. New optimisation constraints are derived to account for any PU state changes within the sensing cycle while implementing the proposed duty cycle based detector.

An examination of keystroke dynamics for continuous user authentication

Most current computer systems authorise the user at the start of a session and do not detect whether the current user is still the initial authorised user, a substitute user, or an intruder pretending to be a valid user. Therefore, a system that continuously checks the identity of the user throughout the session is necessary without being intrusive to end-user and/or effectively doing this. Such a system is called a continuous authentication system (CAS). Researchers have applied several approaches for CAS and most of these techniques are based on biometrics. These continuous biometric authentication systems (CBAS) are supplied by user traits and characteristics. One of the main types of biometric is keystroke dynamics which has been widely tried and accepted for providing continuous user authentication. Keystroke dynamics is appealing for many reasons. First, it is less obtrusive, since users will be typing on the computer keyboard anyway. Second, it does not require extra hardware. Finally, keystroke dynamics will be available after the authentication step at the start of the computer session. Currently, there is insufficient research in the CBAS with keystroke dynamics field. To date, most of the existing schemes ignore the continuous authentication scenarios which might affect their practicality in different real world applications. Also, the contemporary CBAS with keystroke dynamics approaches use characters sequences as features that are representative of user typing behavior but their selected features criteria do not guarantee features with strong statistical significance which may cause less accurate statistical user-representation. Furthermore, their selected features do not inherently incorporate user typing behavior. Finally, the existing CBAS that are based on keystroke dynamics are typically dependent on pre-defined user-typing models for continuous authentication. This dependency restricts the systems to authenticate only known users whose typing samples are modelled. This research addresses the previous limitations associated with the existing CBAS schemes by developing a generic model to better identify and understand the characteristics and requirements of each type of CBAS and continuous authentication scenario. Also, the research proposes four statistical-based feature selection techniques that have highest statistical significance and encompasses different user typing behaviors which represent user typing patterns effectively. Finally, the research proposes the user-independent threshold approach that is able to authenticate a user accurately without needing any predefined user typing model a-priori. Also, we enhance the technique to detect the impostor or intruder who may take over during the entire computer session.

Subcarrier pairing and relay assignment with improved user fairness in a multi-user cooperative OFDM system

We consider a joint relay selection and subcarrier allocation problem that minimizes the total system power for a multi-user, multi-relay and single source cooperative OFDM based two hop system. The system is constrained to all users having a specific subcarrier requirement (user fairness). However no specific fairness constraints for relays are considered. To ensure the optimum power allocation, the subcarriers in two hops are paired with each other. We obtain an optimal subcarrier allocation for the single user case using a similar method to what is described in [1] and modify the algorithm for multiuser scenario. Although the optimality is not achieved in multiuser case the probability of all users being served fairly is improved significantly with a relatively low cost trade off.

Looking Yonder : Longitudinal Research and Evaluation of the Yonder Participative Arts-based Learning Model Report 2010-2012

Since 2007, KITE Arts Education Program @ QPAC has been engaged in a series of arts and drama-based experiences for students in selected primary schools on the edges of Brisbane and in regional Queensland. The in-school workshop experiences of the program have culminated in a performance by the children for their school community, parents and carers at the Queensland Performing Arts Centre or a regional cultural venue. In conducting an analysis of the Yonder project, the researcher aimed to provide evidence of outcomes brought about through participation by schools, school staff, students and their communities in the Yonder project. To develop longitudinal data project initiators, participants were interviewed at six-monthly intervals to establish patterns of engagement and participation. The report analyses arts-based workshops conducted by the teacher artist in edge-city Brisbane and a regional centre; interviews with teachers and school administrators from the participating schools; interviews with teacher artist and professional artists; interviews with community partners; teacher professional development workshops; community-based workshops; performance outcomes that were the culminating events of the workshop program; student work samples and student reflections on the program. This document covers data and project outputs from February 2010 to July 2012. There have been five iterations of the Yonder project since its commencement in mid-2009 — three in regional Queensland (February–April 2010; February–May 2011; February–May 2012) and two in edge-city1 Brisbane (July–September 2010; August–October 2011). This report is a result of a research partnership between Queensland Performing Arts Centre and Queensland University of Technology (QUT) Creative Industries Faculty(Drama).

Nifty after fifty: evaluation of a physical activity directory for older people

Objective: To develop a physical activity directory (PAD) for Brisbane people over the age of 50 years for distribution by two methods (given or requested), and to determine its effectiveness in raising awareness and encouraging older people to participate in local physical activity options. Methods: Baseline demographic data and stage of change was collected from 224 participants who received the directory. Participants were interviewed by telephone six weeks later to determine their use of the directory on a number of dimensions. Results: Most participants interviewed at follow-up remembered reading the directory. Participants who requested the directory were significantly more likely than those who were given it to: be contemplators, read the directory, plan to ring a number, plan to attend a class, and to share the directory with others. Participants who were contemplators were significantly more likely to have participated in physical activity of their own and rang a number from the directory. The directory increased over half the participants' awareness of local physical activity options, yet only 7% reported ringing a number and 15% reported doing their own physical activity. Conclusions: The directory was more effective in raising awareness about physical activity options than encouraging people to participate in physical activity, and participants with short-term plans to be more active were more likely to have used the directory. Implications: The directory, even when linked with other services, raises awareness about physical activity options, but has minimal short-term influence on participation.

Reaching high-risk adolescents : a process evaluation of a school based injury prevention program

High-risk adolescents are a population most vulnerable to harm from injury due to increased engagement in risk taking behaviour. There is a gap in the literature regarding how universal school based injury prevention programs apply to high-risk adolescents. This study involves a component of the process evaluation of a school based injury prevention program, as it relates to high-risk adolescents (13-14 years)...

Community development with survivors of torture and trauma : an evaluation framework

Despite an increase in community development initiatives in refugee contexts, there is a lack of evaluation frameworks to assess the effectiveness of interventions in the recovery of refugee communities. In response to this gap, the Forum of Australian Services for Survivors of Torture and Trauma has developed an evaluation framework in consultation with refugee client groups and agencies' staff members. This paper contextualizes the goals, principles and strategies of services implementing community development initiatives with torture and trauma survivors and describes the process of developing the framework within a participatory action approach. Both outcome evaluation and process evaluation are discussed, and examples of the framework are presented. Community development agencies and professionals working with survivors of torture and trauma can play a significant role by fostering community empowerment through evaluation.

Evaluation of implementation of a healthy food and drink supply strategy throughout the whole school environment in Queensland state schools, Australia

BACKGROUND/OBJECTIVES: This paper reports on the evaluation of the Smart Choices healthy food and drink supply strategy for Queensland schools (Smart Choices) implementation across the whole school environment in state government primary and secondary schools in Queensland, Australia. SUBJECTS/METHODS: Three concurrent surveys using different methods for each group of stakeholders that targeted all 1275 school Principals, all 1258 Parent and Citizens’ Associations (P&Cs) and a random sample of 526 tuckshop convenors throughout Queensland. Nine hundred and seventy-three Principals, 598 P&Cs and 513 tuckshop convenors participated with response rates of 78%, 48% and 98%, respectively. RESULTS: Nearly all Principals (97%), P&Cs (99%) and tuckshop convenors (97%) reported that their school tuckshop had implemented Smart Choices. The majority of Principals and P&Cs reported implementation, respectively, in: school breakfast programs (98 and 92%); vending machine stock (94 and 83%); vending machine advertising (85 and 84%); school events (87 and 88%); school sporting events (81 and 80%); sponsorship and advertising (93 and 84%); fundraising events (80 and 84%); and sporting clubs (73 and 75%). Implementation in curriculum activities, classroom rewards and class parties was reported, respectively, by 97%, 86% and 75% of Principals. Respondents also reported very high levels of understanding of Smart Choices and engagement of the school community. CONCLUSIONS: The results demonstrated that food supply interventions to promote nutrition across all domains of the school environment can be implemented successfully.