Reframe QUT's evaluation framework : a case study of planning, policy and positioning leading to educational best practice

Every university in Australia has a set of policies that guide the institution in its educational practices, however, the policies are often developed in isolation to each other. Now imagine a space where policies are evidence-based, refined annually, cohesively interrelated, and meet stakeholders’ needs. Is this happenstance or the result of good planning? Culturally, Queensland University of Technology (QUT) is a risk-averse institution that takes pride in its financial solvency and is always keen to know “how are we going?” With a twenty-year history of annual reporting that assures the quality of course performance through multiple lines of evidence, QUT’s Learning and Teaching Unit went one step further and strategically aligned a suite of policies that take into consideration the needs of their stakeholders, collaborate with other areas across the institution and use multiple lines of evidence to inform curriculum decision-making. In QUT’s experience, strategic planning can lead to policy that is designed to meet stakeholders’ needs, not manage them; where decision-making is supported by evidence, not rhetoric; where all feedback is incorporated, not ignored; and where policies are cohesively interrelated, not isolated. While many may call this ‘policy nirvana’, QUT has positioned itself to demonstrate good educational practice through Reframe, its evaluation framework. In this case, best practice was achieved through the application of a theory of change and a design-led logic model that allows for transition to other institutions with different cultural specificity. The evaluation approach follows Seldin’s (2003) notion to offer depth and breadth to the evaluation framework along with Berk’s (2005) concept of multiple lines of evidence. In summary, this paper offers university executives, academics, planning and quality staff an opportunity to understand the critical steps that lead to strategic planning and design of evidence-based educational policy that positions a university for best practice in learning and teaching.

Course evidence matrix : a collaborative evaluation using multiple lines of evidence

Welcome to the Evaluation of course matrix. This matrix is designed for highly qualified discipline experts to evaluate their course, major or unit in a systemic manner. The primary purpose of the Evaluation of course matrix is to provide a tool that a group of academic staff at universities can collaboratively review the assessment within a course, major or unit annually. The annual review will result in you being ready for an external curricula review at any point in time. This tool is designed for use in a workshop format with one, two or more academic staff, and will lead to an action plan for implementation. I hope you find this tool useful in your assessment review.

Illuminative evaluation as a method applied to Australian Government policy borrowing and implementation in higher education

Any government deciding to invoke widespread change in its higher education sector through implementation of new policies impacts on every institution and all staff and students, often in both the time taken up and the heightened emotions caused. The central phenomenon that this study addresses is the process and consequences of policy changes in higher education in Australia. The aim of this article is to record the research design through the perspective (evaluation research), theoretical framework (program evaluation) and methods (content analysis, descriptive statistical analysis and bibliometric analysis) applied to the investigation of the 2003 federal government higher education reform package. This approach allows both the intended and unintended consequences arising from the policy implementation of three national initiatives focused on learning and teaching in higher education in Australia to surface. As a result, this program evaluation, also known in some disciplines as policy implementation analysis, will demonstrate the applicability of illuminative evaluation as a methodology and reinforce how program evaluation will assist and advise future government reform and policy implementation, and will serve as a legacy for future evaluative research.Any government deciding to invoke widespread change in its higher education sector through implementation of new policies impacts on every institution and all staff and students, often in both the time taken up and the heightened emotions caused. The central phenomenon that this study addresses is the process and consequences of policy changes in higher education in Australia. The aim of this article is to record the research design through the perspective (evaluation research), theoretical framework (program evaluation) and methods (content analysis, descriptive statistical analysis and bibliometric analysis) applied to the investigation of the 2003 federal government higher education reform package. This approach allows both the intended and unintended consequences arising from the policy implementation of three national initiatives focused on learning and teaching in higher education in Australia to surface. As a result, this program evaluation, also known in some disciplines as policy implementation analysis, will demonstrate the applicability of illuminative evaluation as a methodology and reinforce how program evaluation will assist and advise future government reform and policy implementation, and will serve as a legacy for future evaluative research.

Quality assurance of assessment and moderation discourses involving sessional staff

Quality assurance is a major agenda in tertiary education. The casualisation of academic work, especially in teaching, is also a quality assurance issue. Casual or sessional staff members teach and assess more than 50% of all university courses in Australia, and yet the research in relation to the role sessional staff play in quality assurance of student assessment outcomes is scarce. Moderation processes are a pivotal part of robust quality assurance measures. Drawing upon previous work surrounding four discourses of moderation, this pilot project reports the results of research into the role and impact of sessional staff in moderation processes at tertiary level. Qualitative data were gathered through focus interviews. Results, in the form of various moderation discourses, indicate that sessional staff impact the formal quality assurance processes in numerous ways.

Spotlight on animating spaces: Evaluation of Artslink Queensland's Animating Spaces 2013 & 2014

Animating Spaces 2013-14 Interim Report Artslink Queensland’s Animating Spaces initiative is a three-year statewide project (2013-2015) that aims to revitalise and celebrate non-traditional, significant, and unusual spaces within fifteen regional Queensland communities. After the completion of two years of Animating Spaces this is the first public interim report highlighting outcomes to date. A final evaluation report will be collated in early 2016 that will capture the Animating Spaces project in its entirety. Developed by Queensland University of Technology evaluation team, Professor Helen Klaebe and Dr. Elizabeth Ellison with the extraordinary assistance of Artslink Queensland staff, in particular Kerryanne Farrer.

The development and evaluation of an E-learning package to enhance interprofessional learning

Experience’s of VLE - In some cases was used as little more than a “dumping” ground for lecture/tutorial material. - Allowed easy access to material for staff and students alike. - Attempted use as an interactive teaching and learning platform predominately from a distance learning perspective. Students worked very much in isolation. - Support from Academic and Clinical staff adhoc at times.

Development and evaluation of training in culturally specific screening and brief intervention for hospital patients with alcohol-related injuries

Objective To evaluate health practitioners’ confidence and knowledge of alcohol screening, brief intervention and referral after training in a culturally adapted intervention on alcohol misuse and well-being issues for trauma patients. Design Mixed methods, involving semi-structured interviews at baseline and a post-workshop questionnaire. Setting: Targeted acute care within a remote area major tertiary referral hospital. Participants Ten key informants and 69 questionnaire respondents from relevant community services and hospital-based health care professionals. Intervention Screening and brief intervention training workshops and resources for 59 hospital staff. Main outcome measures Self-reported staff knowledge of alcohol screening, brief intervention and referral, and satisfaction with workshop content and format. Results After training, 44% of participants reported being motivated to implement alcohol screening and intervention. Satisfaction with training was high, and most participants reported that their knowledge of screening and brief intervention was improved. Conclusion Targeted educational interventions can improve the knowledge and confidence of inpatient staff who manage patients at high risk of alcohol use disorder. Further research is needed to determine the duration of the effect and influence on practice behaviour. Ongoing integrated training, linked with systemic support and established quality improvement processes, is required to facilitate sustained change and widespread dissemination.

Evaluation of an intervention for patients with alcohol-related injuries: results of a mixed methods study

Objective: To explore the effect of education and training on the delivery of alcohol screening and brief intervention and referral to high-risk patients in a hospital setting. Main outcome measures included; delivery of training; practice change in relation to staff performing alcohol screening, brief intervention and referrals. Methods: Observational study design using mixed methods set in a tertiary referral hospital. Pre-post assessment of medical records and semi-structured interviews with key informants. Results: Routine screening for substance misuse (9% pre / 71.4% post) and wellbeing concerns (6.6% pre / 15 % post) was more frequent following the introduction of resources and staff participation in educational workshops. There was no evidence of a concomitant increase in delivery of brief intervention or referrals to services. Implementation challenges, including time constraints and staff attitudes, and enablers such as collaboration and visible pathways, were identified. Conclusion: Rates of patient screening increased, however barriers to delivery of brief intervention and referrals remained. Implementation strategies targeting specific barriers and enablers to introducing interventions are both required to improve the application of secondary prevention for patients in acute settings. Implications: Educational training, formalised liaison between services, systematised early intervention protocols, and continuous quality improvement processes will progress service delivery in this area.

Implementing music therapy on an adolescent inpatient unit: A mixed-methods evaluation of acceptability, experience of participation and perceived impact

Objective: We aimed to assess the feasibility of delivering a music therapy program on adolescent psychiatric wards. Method: We undertook a mixed-methods evaluation of a pilot program. Various active and receptive techniques were employed in group music therapy sessions delivered as part of a structured clinical program. Data collected in interviews with participants and staff and feedback questionnaires were thematically and descriptively analysed and triangulated. Results: Data from 62 questionnaires returned by 43 patients who took part in 16 music therapy sessions, and seven staff, evidenced strong support for music therapy. Patients typically reported experiencing sessions as relaxing, comforting, uplifting, and empowering; >90% would participate by choice and use music therapeutically in the future. Staff endorsed music therapy as valuable therapeutically, reporting that patients engaged enthusiastically and identified sessions as improving their own moods and ward milieu. Conclusions: Integration of music therapy in inpatient treatment of adolescents is feasible and acceptable, and is valued by staff and patients as a complement to ‘talking therapies’. Participation is enjoyed and associated with outcomes including improvement in mood, expression of feelings and social engagement consistent with recovery.

Educators’ perspectives about how older hospital patients can engage in a falls prevention education programme: A qualitative process evaluation

- Objectives Falls are the most frequent adverse event reported in hospitals. Patient and staff education delivered by trained educators significantly reduced falls and injurious falls in an older rehabilitation population. The purpose of the study was to explore the educators’ perspectives of delivering the education and to conceptualise how the programme worked to prevent falls among older patients who received the education. - Design A qualitative exploratory study. - Methods Data were gathered from three sources: conducting a focus group and an interview (n=10 educators), written educator notes and reflective researcher field notes based on interactions with the educators during the primary study. The educators delivered the programme on eight rehabilitation wards for periods of between 10 and 40 weeks. They provided older patients with individualised education to engage in falls prevention and provided staff with education to support patient actions. Data were thematically analysed and presented using a conceptual framework. - Results Falls prevention education led to mutual understanding between staff and patients which assisted patients to engage in falls prevention behaviours. Mutual understanding was derived from the following observations: the educators perceived that they could facilitate an effective three-way interaction between staff actions, patient actions and the ward environment which led to behaviour change on the wards. This included engaging with staff and patients, and assisting them to reconcile differing perspectives about falls prevention behaviours. - Conclusions Individualised falls prevention education effectively provides patients who receive it with the capability and motivation to develop and undertake behavioural strategies that reduce their falls, if supported by staff and the ward environment.

Design challenges for electronic medication administration record systems in residential aged care facilities: A formative evaluation

Introduction Electronic medication administration record (eMAR) systems are promoted as a potential intervention to enhance medication safety in residential aged care facilities (RACFs). The purpose of this study was to conduct an in-practice evaluation of an eMAR being piloted in one Australian RACF before its roll out, and to provide recommendations for system improvements. Methods A multidisciplinary team conducted direct observations of workflow (n=34 hours) in the RACF site and the community pharmacy. Semi-structured interviews (n=5) with RACF staff and the community pharmacist were conducted to investigate their views of the eMAR system. Data were analysed using a grounded theory approach to identify challenges associated with the design of the eMAR system. Results The current eMAR system does not offer an end-to-end solution for medication management. Many steps, including prescribing by doctors and communication with the community pharmacist, are still performed manually using paper charts and fax machines. Five major challenges associated with the design of eMAR system were identified: limited interactivity; inadequate flexibility; problems related to information layout and semantics; the lack of relevant decision support; and system maintenance issues.We suggest recommendations to improve the design of the eMAR system and to optimize existing workflows. Discussion Immediate value can be achieved by improving the system interactivity, reducing inconsistencies in data entry design and offering dedicated organisational support to minimise connectivity issues. Longer-term benefits can be achieved by adding decision support features and establishing system interoperability requirements with stakeholder groups (e.g. community pharmacies) prior to system roll out. In-practice evaluations of technologies like eMAR system have great value in identifying design weaknesses which inhibit optimal system use.

An Evaluation of the area stratification used for sampling tunas in the eastern Pacific Ocean and implications for estimating total annual catches

The Inter-American Tropical Tuna Commission (IATTC) staff has been sampling the size distributions of tunas in the eastern Pacific Ocean (EPO) since 1954, and the species composition of the catches since 2000. The IATTC staff use the data from the species composition samples, in conjunction with observer and/or logbook data, and unloading data from the canneries to estimate the total annual catches of yellowfin (Thunnus albacares), skipjack (Katsuwonus pelamis), and bigeye (Thunnus obesus) tunas. These sample data are collected based on a stratified sampling design. I propose an update of the stratification of the EPO into more homogenous areas in order to reduce the variance in the estimates of the total annual catches and incorporate the geographical shifts resulting from the expansion of the floating-object fishery during the 1990s. The sampling model used by the IATTC is a stratified two-stage (cluster) random sampling design with first stage units varying (unequal) in size. The strata are month, area, and set type. Wells, the first cluster stage, are selected to be sampled only if all of the fish were caught in the same month, same area, and same set type. Fish, the second cluster stage, are sampled for lengths, and independently, for species composition of the catch. The EPO is divided into 13 sampling areas, which were defined in 1968, based on the catch distributions of yellowfin and skipjack tunas. This area stratification does not reflect the multi-species, multi-set-type fishery of today. In order to define more homogenous areas, I used agglomerative cluster analysis to look for groupings of the size data and the catch and effort data for 2000–2006. I plotted the results from both datasets against the IATTC Sampling Areas, and then created new areas. I also used the results of the cluster analysis to update the substitution scheme for strata with catch, but no sample. I then calculated the total annual catch (and variance) by species by stratifying the data into new Proposed Sampling Areas and compared the results to those reported by the IATTC. Results showed that re-stratifying the areas produced smaller variances of the catch estimates for some species in some years, but the results were not significant.

NHS Wales e-library portal evaluation (For Informing Healthcare Strategy implementation programme)

Thomas, R. & Urquhart, C. NHS Wales e-library portal evaluation. (For Informing Healthcare Strategy implementation programme). Aberystwyth: Department of Information Studies, University of Wales Aberystwyth Follow-on to NHS Wales User Needs study Sponsorship: Informing Healthcare, NHS Wales

Training needs analysis of healthcare library staff, undertaken for South Yorkshire Workforce Development Confederation

Urquhart, C., Durbin, J. & Spink, S. (2004). Training needs analysis of healthcare library staff, undertaken for South Yorkshire Workforce Development Confederation. Aberystwyth: Department of Information Studies, University of Wales Aberystwyth. Sponsorship: South Yorkshire WDC (NHS)

Evaluation of resources to support production of high quality health information for patients and the public. Final report for NHS Research Outputs Programme

Durbin, J., Urquhart, C. & Yeoman, A. (2003). Evaluation of resources to support production of high quality health information for patients and the public. Final report for NHS Research Outputs Programme. Aberystwyth: Department of Information Studies, University of Wales Aberystwyth. Sponsorship: Department of Health