Development and evaluation of a program to improve clinician and patient communication during a telehealth consultation : C.R.I.S.P. Telehealth

Introduction: The delivery of health care in the 21st century will look like no other in the past. The fast paced technological advances that are being made will need to transition from the information age into clinical practice. The phenomenon of e-Health is the over-arching form of information technology and telehealth is one arm of that phenomenon. The uptake of telehealth both in Australia and overseas, has changed the face of health service delivery to many rural and remote communities for the better, removing what is known as the tyranny of distance. Many studies have evaluated the satisfaction and cost-benefit analysis of telehealth across the organisational aspects as well as the various adaptations of clinical pathways and this is the predominant focus of most studies published to date. However, whilst comments have been made by many researchers about the need to improve and attend to the communication and relationship building aspects of telehealth no studies have examined this further. The aim of this study was to identify the patient and clinician experiences, concerns, behaviours and perceptions of the telehealth interaction and develop a training tool to assist these clinicians to improve their interaction skills. Methods: A mixed methods design combining quantitative (survey analysis and data coding) and qualitative (interview analysis) approaches was adopted. This study utilised four phases to firstly qualitatively explore the needs of clients (patients) and clinicians within a telehealth consultation then designed, developed, piloted and quantitatively and qualitatively evaluated the telehealth communication training program. Qualitative data was collected and analysed during Phase 1 of this study to describe and define the missing 'communication and rapport building' aspects within telehealth. This data was then utilised to develop a self-paced communication training program that enhanced clinicians existing skills, which comprised of Phase 2 of this study to develop the interactive program. Phase 3 included evaluating the training program with 26 clinicians and results were recorded pre and post training, whilst phase 4 was the pilot for future recommendations of this training program using a patient group within a Queensland Health setting at two rural hospitals. Results: Comparisons of pre and post training data on 1) Effective communication styles, 2) Involvement in communication training package, 3) satisfaction pre and post training, and 4) health outcomes pre and post training indicated that there were differences between pre and post training in relation to effective communication style, increased satisfaction and no difference in health outcomes between pre and post training for this patient group. The post training results revealed over half of the participants (N= 17, 65%) were more responsive to non-verbal cues and were better able to reflect and respond to looks of anxiousness and confusion from a 'patient' within a telehealth consultation. It was also found that during post training evaluations, clinicians had enhanced their therapeutic communication with greater detail to their own body postures, eye contact and presentation. There was greater time spent looking at the 'patient' with an increase of 35 second intervals of direct eye contact and less time spent looking down at paperwork which decreased by 20 seconds. Overall 73% of the clinicians were satisfied with the training program and 61% strongly agreed that they recognised areas of their communication that needed improving during a telehealth consultation. For the patient group there was significant difference post training in rapport with a mean score from 42 (SD = 28, n = 27) to 48 (SD = 5.9, n = 24). For communication comfort of the patient group there was a significant difference between the pre and post training scores t(10) = 27.9, p = .002, which meant that overall the patients felt less inhibited whilst talking to the clinicians and more understood. Conclusion: The aim of this study was to explore the characteristics of good patient-clinician communication and unmet training needs for telehealth consultations. The study developed a training program that was specific for telehealth consultations and not dependent on a 'trainer' to deliver the content. In light of the existing literature this is a first of its kind and a valuable contribution to the research on this topic. It was found that the training program was effective in improving the clinician's communication style and increased the satisfaction of patient's within an e-health environment. This study has identified some historical myths that telehealth cannot be part of empathic patient centred care due to its technology tag.

Efficacy of proxy definitions for identification of fatigue/sleep-related crashes : an Australian evaluation

Fatigue/sleepiness is recognised as an important contributory factor in fatal and serious injury road traffic incidents (RTIs), however, identifying fatigue/sleepiness as a causal factor remains an uncertain science. Within Australia attending police officers at a RTI report the causal factors; one option is fatigue/sleepiness. In some Australian jurisdictions police incident databases are subject to post hoc analysis using a proxy definition for fatigue/sleepiness. This secondary analysis identifies further RTIs caused by fatigue/sleepiness not initially identified by attending officers. The current study investigates the efficacy of such proxy definitions for attributing fatigue/sleepiness as a RTI causal factor. Over 1600 Australian drivers were surveyed regarding their experience and involvement in fatigue/sleep-related RTIs and near-misses during the past five years. Driving while fatigued/sleepy had been experienced by the majority of participants (66.0% of participants). Fatigue/sleep-related near misses were reported by 19.1% of participants, with 2.4% being involved in a fatigue/sleep-related RTI. Examination of the characteristics for the most recent event (either a near miss or crash) found that the largest proportion of incidents (28.0%) occurred when commuting to or from work, followed by social activities (25.1%), holiday travel (19.8%), or for work purposes (10.1%). The fatigue/sleep related RTI and near-miss experience of a representative sample of Australian drivers does not reflect the proxy definitions used for fatigue/sleepiness identification. In particular those RTIs that occur in urban areas and at slow speeds may not be identified. While important to have a strategy for identifying fatigue/sleepiness related RTIs proxy measures appear best suited to identifying specific subsets of such RTIs.

Anatomic fitting of total artificial hearts for in vivo evaluation

Successful anatomic fitting of a total artificial heart (TAH) is vital to achieve optimal pump hemodynamics after device implantation. Although many anatomic fitting studies have been completed in humans prior to clinical trials, few reports exist that detail the experience in animals for in vivo device evaluation. Optimal hemodynamics are crucial throughout the in vivo phase to direct design iterations and ultimately validate device performance prior to pivotal human trials. In vivo evaluation in a sheep model allows a realistically sized representation of a smaller patient, for which smaller third-generation TAHs have the potential to treat. Our study aimed to assess the anatomic fit of a single device rotary TAH in sheep prior to animal trials and to use the data to develop a threedimensional, computer-aided design (CAD)-operated anatomic fitting tool for future TAH development. Following excision of the native ventricles above the atrio-ventricular groove, a prototype TAH was inserted within the chest cavity of six sheep (28–40 kg).Adjustable rods representing inlet and outlet conduits were oriented toward the center of each atrial chamber and the great vessels, with conduit lengths and angles recorded for future analysis. A threedimensional, CAD-operated anatomic fitting tool was then developed, based on the results of this study, and used to determine the inflow and outflow conduit orientation of the TAH. The mean diameters of the sheep left atrium, right atrium, aorta, and pulmonary artery were 39, 33, 12, and 11 mm, respectively. The center-to-center distance and outer-edge-to-outer-edge distance between the atria, found to be 39 ± 9 mm and 72 ± 17 mm in this study, were identified as the most critical geometries for successful TAH connection. This geometric constraint restricts the maximum separation allowable between left and right inlet ports of a TAH to ensure successful alignment within the available atrial circumference.

Evaluation of inflow cannulation site for implantation of right-sided rotary ventricular assist device

Right heart dysfunction is one of the most serious complications following implantation of a left ventricular assist device (LVAD), often leading to the requirement for short or long term right ventricular support (RVAD). The inflow cannulation site induces major haemodynamic changes and so there is a need to optimize the site used depending on the patient's condition. Therefore, this study evaluated and compared the haemodynamic influence of right atrial (RAC) and right ventricular (RVC) inflow cannulation sites. An in-vitro, variable heart failure, mock circulation loop was used to compare RAC and RVC in mild and severe biventricular heart failure (BHF) conditions. In the severe BHF condition, higher ventricular ejection fraction (RAC: 13.6%, RVC: 32.7%) and thus improved heart chamber and RVAD washout was observed with RVC, which suggested this strategy might be preferable for long term support (ie. bridge to transplant or destination therapy) to reduce the risk of thrombus formation. In the mild BHF condition, higher pulmonary valve flow (RAC: 3.33 L/min, RVC: 1.97 L/min) and lower right ventricular stroke work (RAC: 0.10 W, RVC: 0.13 W) and volumes were recorded with RAC. These results indicate an improved potential for myocardial recovery, thus RAC should be chosen in this condition. This in-vitro study suggests that RVAD inflow cannulation site should be chosen on a patient-specific basis with a view to the support strategy to promote myocardial recovery or reduce the risk of long-term complications.

Synthesis and evaluation of resveratrol-based nitroxides as potential treatments for hypertension

The synthesis and evaluation of novel resveratrol-based nitroxides have been explored for the potential treatment of hypertension. New methodology for the direct aryl iodination of isoindoline and isoindoline nitroxide using periodic acid and potassium iodide in concentrated sulphuric acid was developed. Diiodinated tetramethyl and tetraethyl isoindolines and a tetramethyl isoindoline nitroxide were prepared in excellent yields (70 – 82%). A diiodinated tetraethyl isoindoline nitroxide was generated from the corresponding nitroxide in modest yield (37%) alongside iodinated nitrones. The mono-iodinated species were also generated in modest yields (34 – 48%). Incorporation of the nitroxide unit into the structure of resveratrol was achieved using palladium-catalysed Heck coupling. Use of the previously prepared iodo products 5-iodo-1,1,3,3-tetramethylisoindolin-2-yloyl 18 and 5,6-diiodo-1,1,3,3-tetramethylisoindolin-2-yloyl 22 gave resveratrol nitroxides 12 and 13 in yields of 50% (optimized) and 1.6% respectively. Preliminary evaluation of the resveratrol analogue 12 as a treatment for hypertension was undertaken in the DOCA-salt rat model. A reduction in systolic blood pressure as well as alleviation of ventricular hypertrophy was observed. A larger study involving the DOCA salt rats is currently in progress.

A pre-post evaluation of an ambulatory nutrition support service for malnourished patients post hospital discharge: a pilot study

Introduction Malnutrition is common among hospitalised patients, with poor follow-up of nutrition support post-discharge. Published studies on the efficacy of ambulatory nutrition support (ANS) for malnourished patients post-discharge are scarce. The aims of this study were to evaluate the rate of dietetics follow-up of malnourished patients post-discharge, before (2008) and after (2010) implementation of a new ANS service, and to evaluate nutritional outcomes post-implementation. Materials and Methods Consecutive samples of 261 (2008) and 163 (2010) adult inpatients referred to dietetics and assessed as malnourished using Subjective Global Assessment (SGA) were enrolled. All subjects received inpatient nutrition intervention and dietetic outpatient clinic follow-up appointments. For the 2010 cohort, ANS was initiated to provide telephone follow-up and home visits for patients who failed to attend the outpatient clinic. Subjective Global Assessment, body weight, quality of life (EQ-5D VAS) and handgrip strength were measured at baseline and five months post-discharge. Paired t-test was used to compare pre- and post-intervention results. Results In 2008, only 15% of patients returned for follow-up with a dietitian within four months post-discharge. After implementation of ANS in 2010, the follow-up rate was 100%. Mean weight improved from 44.0 ± 8.5kg to 46.3 ± 9.6kg, EQ-5D VAS from 61.2 ± 19.8 to 71.6 ± 17.4 and handgrip strength from 15.1 ± 7.1 kg force to 17.5 ± 8.5 kg force; p<0.001 for all. Seventy-four percent of patients improved in SGA score. Conclusion Ambulatory nutrition support resulted in significant improvements in follow-up rate, nutritional status and quality of life of malnourished patients post-discharge.

Behavioral evaluation of eight rat lines selected for high and low anxiety-related responses

Anxiety traits can be stable and permanent characteristics of an individual across time that is less susceptible of influences by a particular situation. One way to study trait anxiety in an experimental context is through the use of rat lines, selected according to contrasting phenotypes of fear and anxiety. It is not clear whether the behavioral differences between two contrasting rat lines in one given anxiety test are also present in others paradigms of state anxiety. Here, we examine the extent to which multiple anxiety traits generalize across selected animal lines originally selected for a single anxiety trait. We review the behavioral results available in the literature of eight rat genetic models of trait anxiety - namely Maudsley Reactive and Non-reactive rats, Floripa H and L rats, Tsukuba High and Low Emotional rats, High and Low Anxiety-related rats, High and Low Ultrasonic Vocalization rats, Roman High and Low Avoidance rats, Syracuse High and Low Avoidance rats, and Carioca High and Low Conditioned Freezing rats - across 11 behavioral paradigms of innate anxiety or aversive learning frequently used in the experimental setting. We observed both convergence and divergence of behavioral responses in these selected lines across the 11 paradigms. We find that predisposition for specific anxiety traits will usually be generalized to other anxiety provoking stimuli. However this generalization is not observed across all genetic models indicating some unique trait and state interactions. Genetic models of enhanced-anxiety related responses are beginning to help define how anxiety can manifest differently depending on the underlying traits and the current environmentally induced state.

Measurement as legitimacy versus legitimacy of measures : Performance evaluation of social enterprise

Purpose The purpose of this paper is to review the growing emphasis on quantifiable performance measures such as social return on investment (SROI) in third sector organisations – specifically, social enterprise – through a legitimacy theory lens. It then examines what social enterprises value (i.e. consider important) in terms of performance evaluation, using a case study approach. Design/methodology/approach Case studies involving interviews, documentary analysis, and observation, of three social enterprises at different life-cycle stages with different funding structures, were constructed to consider “what measures matter” from a practitioner's perspective. Findings Findings highlight a priority on quality outcomes and impacts in primarily qualitative terms to evaluate performance. Further, there is a noticeable lack of emphasis on financial measures other than basic access to financial resources to continue pursuing social goals. Social implications The practical challenges faced by social enterprises – many of which are small to medium sized – in evaluating performance and by implication organisational legitimacy are contrasted with measures such as SROI which are resource intensive and have inherent methodological limitations. Hence, findings suggest the limited and valuable resources of social enterprises would be better allocated towards documenting the actual outcomes and impacts as a first step, in order to evaluate social and financial performance in terms appropriate to each objective, in order to demonstrate organisational legitimacy. Originality/value Findings distinguish between processes which may hold symbolic legitimacy for select stakeholder groups, and processes which hold substantive, cognitive legitimacy for stakeholders more broadly, in the under-researched context of social enterprise.

Identifying leadership practices for mentoring preservice teachers

There are many forms of leadership and concepts of school leadership have evolved significantly over the last few decades. Mentoring is a form of leadership, where the classroom teacher (mentor) leads and guides the preservice teacher towards advancing teaching practices. What do school executives identify as their leadership practices and what leadership practices have inspired them? This study uses a five-part Likert scale survey with extended written responses that were coded into themes. These participants indicated they had leadership potential, which they associated with being organised, passionate and knowledgeable about education, interpersonally-skilled to build relationships, and visionary with action plans for improving education. These practices were also identified by participants as inspiring practices from leaders they knew. Generally, these participants perceived themselves as transformational leaders. Transformational practices associated with individualized consideration, intellectual stimulation, inspirational motivation, and idealized influences were agreed upon by 80% or more of the participants. Mentors need to understand inspiring leadership practices and identify their own leadership practices that may lead towards reflection on practice and, hence, a way to make educationally-sound changes in leadership behaviour.

Development and formative evaluation of a motorcycle rider training intervention to address risk taking

The need to address on-road motorcycle safety in Australia is important due to the disproportionately high percentage of riders and pillions killed and injured each year. One approach to preventing motorcycle-related injury is through training and education. However, motorcycle rider training lacks empirical support as an effective road safety countermeasure to reduce crash involvement. Previous reviews have highlighted that risk-taking is a contributing factor in many motorcycle crashes, rather than merely a lack of vehicle-control skills (Haworth & Mulvihill, 2005; Jonah, Dawson & Bragg, 1982; Watson et al, 1996). Hence, though the basic vehicle-handling skills and knowledge of road rules that are taught in most traditional motorcycle licence training programs may be seen as an essential condition of safe riding, they do not appear to be sufficient in terms of crash reduction. With this in mind there is considerable scope for the improvement of program focus and content for rider training and education. This program of research examined an existing traditional pre-licence motorcycle rider training program and formatively evaluated the addition of a new classroom-based module to address risky riding; the Three Steps to Safer Riding program. The pilot program was delivered in the real world context of the Q-Ride motorcycle licensing system in the state of Queensland, Australia. Three studies were conducted as part of the program of research: Study 1, a qualitative investigation of delivery practices and student learning needs in an existing rider training course; Study 2, an investigation of the extent to which an existing motorcycle rider training course addressed risky riding attitudes and motives; and Study 3, a formative evaluation of the new program. A literature review as well as the investigation of learning needs for motorcyclists in Study 1 aimed to inform the initial planning and development of the Three Steps to Safer Riding program. Findings from Study 1 suggested that the training delivery protocols used by the industry partner training organisation were consistent with a learner-centred approach and largely met the learning needs of trainee riders. However, it also found that information from the course needs to be reinforced by on-road experiences for some riders once licensed and that personal meaning for training information was not fully gained until some riding experience had been obtained. While this research informed the planning and development of the new program, a project team of academics and industry experts were responsible for the formulation of the final program. Study 2 and Study 3 were conducted for the purpose of formative evaluation and program refinement. Study 2 served primarily as a trial to test research protocols and data collection methods with the industry partner organisation and, importantly, also served to gather comparison data for the pilot program which was implemented with the same rider training organisation. Findings from Study 2 suggested that the existing training program of the partner organisation generally had a positive (albeit small) effect on safety in terms of influencing attitudes to risk taking, the propensity for thrill seeking, and intentions to engage in future risky riding. However, maintenance of these effects over time and the effects on riding behaviour remain unclear due to a low response rate upon follow-up 24 months after licensing. Study 3 was a formative evaluation of the new pilot program to establish program effects and possible areas for improvement. Study 3a examined the short term effects of the intervention pilot on psychosocial factors underpinning risky riding compared to the effects of the standard traditional training program (examined in Study 2). It showed that the course which included the Three Steps to Safer Riding program elicited significantly greater positive attitude change towards road safety than the existing standard licensing course. This effect was found immediately following training, and mean scores for attitudes towards safety were also maintained at the 12 month follow-up. The pilot program also had an immediate effect on other key variables such as risky riding intentions and the propensity for thrill seeking, although not significantly greater than the traditional standard training. A low response rate at the 12 month follow-up unfortunately prevented any firm conclusions being drawn regarding the impact of the pilot program on self-reported risky riding once licensed. Study 3a further showed that the use of intermediate outcomes such as self-reported attitudes and intentions for evaluation purposes provides insights into the mechanisms underpinning risky riding that can be changed by education and training. A multifaceted process evaluation conducted in Study 3b confirmed that the intervention pilot was largely delivered as designed, with course participants also rating most aspects of training delivery highly. The complete program of research contributed to the overall body of knowledge relating to motorcycle rider training, with some potential implications for policy in the area of motorcycle rider licensing. A key finding of the research was that psychosocial influences on risky riding can be shaped by structured education that focuses on awareness raising at a personal level and provides strategies to manage future riding situations. However, the formative evaluation was mainly designed to identify areas of improvement for the Three Steps to Safer Riding program and found several areas of potential refinement to improve future efficacy of the program. This included aspects of program content, program delivery, resource development, and measurement tools. The planned future follow-up of program participants' official crash and traffic offence records over time may lend further support for the application of the program within licensing systems. The findings reported in this thesis offer an initial indication that the Three Steps to Safer Riding is a useful resource to accompany skills-based training programs.

Evaluation of an ambulatory nutrition support service for malnourished patients post hospital discharge

Introduction Malnutrition is common among hospitalised patients, with poor follow-up of nutrition support post-discharge. Published studies on the efficacy of ambulatory nutrition support (ANS) for malnourished patients post-discharge are scarce. The aims of this study were to evaluate the rate of dietetics follow-up of malnourished patients post-discharge, before (2008) and after (2010) implementation of a new ANS service, and to evaluate nutritional outcomes post-implementation. Materials and Methods Consecutive samples of 261 (2008) and 163 (2010) adult inpatients referred to dietetics and assessed as malnourished using Subjective Global Assessment (SGA) were enrolled. All subjects received inpatient nutrition intervention and dietetic outpatient clinic follow-up appointments. For the 2010 cohort, ANS was initiated to provide telephone follow-up and home visits for patients who failed to attend the outpatient clinic. Subjective Global Assessment, body weight, quality of life (EQ-5D VAS) and handgrip strength were measured at baseline and five months post-discharge. Paired t-test was used to compare pre- and post-intervention results. Results In 2008, only 15% of patients returned for follow-up with a dietitian within four months post-discharge. After implementation of ANS in 2010, the follow-up rate was 100%. Mean weight improved from 44.0 ± 8.5kg to 46.3 ± 9.6kg, EQ-5D VAS from 61.2 ± 19.8 to 71.6 ± 17.4 and handgrip strength from 15.1 ± 7.1 kg force to 17.5 ± 8.5 kg force; p<0.001 for all. Seventy-four percent of patients improved in SGA score. Conclusion Ambulatory nutrition support resulted in significant improvements in follow-up rate, nutritional status and quality of life of malnourished patients post-discharge.

Development, implementation and evaluation of a cancer survivor self-management care plan

Aim There is a growing population of people with cancer who experience physiological and psychological effects that persist long after treatment is complete. Interventions that enhance survivors’ self-management abilities might help offset these effects. The aim of this pilot study was to develop, implement and evaluate interventions tailored to assist patients to manage post-treatment health issues effectively. Method In this pre-post intervention cohort study, participants were recruited on completion of cancer treatment. Participants recruited preimplementation, who received usual care, comprised the control group. Participants recruited later formed the intervention group. In the intervention group, the Cancer Care Coordinator developed an individualised, structured Cancer Survivor Self-management Care Plan. Participants were interviewed on completion of treatment (baseline) and at three months. Assessments concerned health needs (CaSUN), self-efficacy in adjusting and coping with cancer and health-related quality of life (FACIT-B or FACT-C). The impact of the intervention was determined by independent t-tests of change scores. Results The intervention (n = 32) and control groups (n = 35) were comparable on demographic and clinical characteristics. Sample mean age was 54 + 10 years. Cancer diagnoses were breast (82%) and colorectal (18%). Statistically significant differences (p < 0.05) indicated improvement in the intervention group for: (a) functional well-being, from the FACIT, (Control: M = −0.69, SE = 0.91; Intervention: M = 3.04, SE = 1.13); and (b) self-efficacy in maintaining social relationships, (Control: M = −0.333, SE = 0.33; Intervention: M = 0.621, SE = 0.27). No significant differences were found in health needs, other subscales of quality of life, the extent and number of strategies used in coping and adjusting to cancer and in other domains of self-efficacy. Conclusions While the results should be interpreted with caution, due to the non-randomised nature of the study and the small sample size, they indicate the potential benefits of tailored self-management interventions warrant further investigation in this context.

Review: "Context : A framework for its Influence on Evaluation Practice (New Directions for Evaluation No. 135)" edited by Rog, D. J.; Fitzpatrick, J. L. and Conner, R. F.

This book offers a framework for the influence of context on evaluation practice and is applied to three case studies: environmental context; indigenous context and political context; and finishes with a process for implementation.

Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

REFRAME: a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.