Measurement as legitimacy versus legitimacy of measures : Performance evaluation of social enterprise

Purpose The purpose of this paper is to review the growing emphasis on quantifiable performance measures such as social return on investment (SROI) in third sector organisations – specifically, social enterprise – through a legitimacy theory lens. It then examines what social enterprises value (i.e. consider important) in terms of performance evaluation, using a case study approach. Design/methodology/approach Case studies involving interviews, documentary analysis, and observation, of three social enterprises at different life-cycle stages with different funding structures, were constructed to consider “what measures matter” from a practitioner's perspective. Findings Findings highlight a priority on quality outcomes and impacts in primarily qualitative terms to evaluate performance. Further, there is a noticeable lack of emphasis on financial measures other than basic access to financial resources to continue pursuing social goals. Social implications The practical challenges faced by social enterprises – many of which are small to medium sized – in evaluating performance and by implication organisational legitimacy are contrasted with measures such as SROI which are resource intensive and have inherent methodological limitations. Hence, findings suggest the limited and valuable resources of social enterprises would be better allocated towards documenting the actual outcomes and impacts as a first step, in order to evaluate social and financial performance in terms appropriate to each objective, in order to demonstrate organisational legitimacy. Originality/value Findings distinguish between processes which may hold symbolic legitimacy for select stakeholder groups, and processes which hold substantive, cognitive legitimacy for stakeholders more broadly, in the under-researched context of social enterprise.

Development and formative evaluation of a motorcycle rider training intervention to address risk taking

The need to address on-road motorcycle safety in Australia is important due to the disproportionately high percentage of riders and pillions killed and injured each year. One approach to preventing motorcycle-related injury is through training and education. However, motorcycle rider training lacks empirical support as an effective road safety countermeasure to reduce crash involvement. Previous reviews have highlighted that risk-taking is a contributing factor in many motorcycle crashes, rather than merely a lack of vehicle-control skills (Haworth & Mulvihill, 2005; Jonah, Dawson & Bragg, 1982; Watson et al, 1996). Hence, though the basic vehicle-handling skills and knowledge of road rules that are taught in most traditional motorcycle licence training programs may be seen as an essential condition of safe riding, they do not appear to be sufficient in terms of crash reduction. With this in mind there is considerable scope for the improvement of program focus and content for rider training and education. This program of research examined an existing traditional pre-licence motorcycle rider training program and formatively evaluated the addition of a new classroom-based module to address risky riding; the Three Steps to Safer Riding program. The pilot program was delivered in the real world context of the Q-Ride motorcycle licensing system in the state of Queensland, Australia. Three studies were conducted as part of the program of research: Study 1, a qualitative investigation of delivery practices and student learning needs in an existing rider training course; Study 2, an investigation of the extent to which an existing motorcycle rider training course addressed risky riding attitudes and motives; and Study 3, a formative evaluation of the new program. A literature review as well as the investigation of learning needs for motorcyclists in Study 1 aimed to inform the initial planning and development of the Three Steps to Safer Riding program. Findings from Study 1 suggested that the training delivery protocols used by the industry partner training organisation were consistent with a learner-centred approach and largely met the learning needs of trainee riders. However, it also found that information from the course needs to be reinforced by on-road experiences for some riders once licensed and that personal meaning for training information was not fully gained until some riding experience had been obtained. While this research informed the planning and development of the new program, a project team of academics and industry experts were responsible for the formulation of the final program. Study 2 and Study 3 were conducted for the purpose of formative evaluation and program refinement. Study 2 served primarily as a trial to test research protocols and data collection methods with the industry partner organisation and, importantly, also served to gather comparison data for the pilot program which was implemented with the same rider training organisation. Findings from Study 2 suggested that the existing training program of the partner organisation generally had a positive (albeit small) effect on safety in terms of influencing attitudes to risk taking, the propensity for thrill seeking, and intentions to engage in future risky riding. However, maintenance of these effects over time and the effects on riding behaviour remain unclear due to a low response rate upon follow-up 24 months after licensing. Study 3 was a formative evaluation of the new pilot program to establish program effects and possible areas for improvement. Study 3a examined the short term effects of the intervention pilot on psychosocial factors underpinning risky riding compared to the effects of the standard traditional training program (examined in Study 2). It showed that the course which included the Three Steps to Safer Riding program elicited significantly greater positive attitude change towards road safety than the existing standard licensing course. This effect was found immediately following training, and mean scores for attitudes towards safety were also maintained at the 12 month follow-up. The pilot program also had an immediate effect on other key variables such as risky riding intentions and the propensity for thrill seeking, although not significantly greater than the traditional standard training. A low response rate at the 12 month follow-up unfortunately prevented any firm conclusions being drawn regarding the impact of the pilot program on self-reported risky riding once licensed. Study 3a further showed that the use of intermediate outcomes such as self-reported attitudes and intentions for evaluation purposes provides insights into the mechanisms underpinning risky riding that can be changed by education and training. A multifaceted process evaluation conducted in Study 3b confirmed that the intervention pilot was largely delivered as designed, with course participants also rating most aspects of training delivery highly. The complete program of research contributed to the overall body of knowledge relating to motorcycle rider training, with some potential implications for policy in the area of motorcycle rider licensing. A key finding of the research was that psychosocial influences on risky riding can be shaped by structured education that focuses on awareness raising at a personal level and provides strategies to manage future riding situations. However, the formative evaluation was mainly designed to identify areas of improvement for the Three Steps to Safer Riding program and found several areas of potential refinement to improve future efficacy of the program. This included aspects of program content, program delivery, resource development, and measurement tools. The planned future follow-up of program participants' official crash and traffic offence records over time may lend further support for the application of the program within licensing systems. The findings reported in this thesis offer an initial indication that the Three Steps to Safer Riding is a useful resource to accompany skills-based training programs.

Evaluation of an ambulatory nutrition support service for malnourished patients post hospital discharge

Introduction Malnutrition is common among hospitalised patients, with poor follow-up of nutrition support post-discharge. Published studies on the efficacy of ambulatory nutrition support (ANS) for malnourished patients post-discharge are scarce. The aims of this study were to evaluate the rate of dietetics follow-up of malnourished patients post-discharge, before (2008) and after (2010) implementation of a new ANS service, and to evaluate nutritional outcomes post-implementation. Materials and Methods Consecutive samples of 261 (2008) and 163 (2010) adult inpatients referred to dietetics and assessed as malnourished using Subjective Global Assessment (SGA) were enrolled. All subjects received inpatient nutrition intervention and dietetic outpatient clinic follow-up appointments. For the 2010 cohort, ANS was initiated to provide telephone follow-up and home visits for patients who failed to attend the outpatient clinic. Subjective Global Assessment, body weight, quality of life (EQ-5D VAS) and handgrip strength were measured at baseline and five months post-discharge. Paired t-test was used to compare pre- and post-intervention results. Results In 2008, only 15% of patients returned for follow-up with a dietitian within four months post-discharge. After implementation of ANS in 2010, the follow-up rate was 100%. Mean weight improved from 44.0 ± 8.5kg to 46.3 ± 9.6kg, EQ-5D VAS from 61.2 ± 19.8 to 71.6 ± 17.4 and handgrip strength from 15.1 ± 7.1 kg force to 17.5 ± 8.5 kg force; p<0.001 for all. Seventy-four percent of patients improved in SGA score. Conclusion Ambulatory nutrition support resulted in significant improvements in follow-up rate, nutritional status and quality of life of malnourished patients post-discharge.

Development, implementation and evaluation of a cancer survivor self-management care plan

Aim There is a growing population of people with cancer who experience physiological and psychological effects that persist long after treatment is complete. Interventions that enhance survivors’ self-management abilities might help offset these effects. The aim of this pilot study was to develop, implement and evaluate interventions tailored to assist patients to manage post-treatment health issues effectively. Method In this pre-post intervention cohort study, participants were recruited on completion of cancer treatment. Participants recruited preimplementation, who received usual care, comprised the control group. Participants recruited later formed the intervention group. In the intervention group, the Cancer Care Coordinator developed an individualised, structured Cancer Survivor Self-management Care Plan. Participants were interviewed on completion of treatment (baseline) and at three months. Assessments concerned health needs (CaSUN), self-efficacy in adjusting and coping with cancer and health-related quality of life (FACIT-B or FACT-C). The impact of the intervention was determined by independent t-tests of change scores. Results The intervention (n = 32) and control groups (n = 35) were comparable on demographic and clinical characteristics. Sample mean age was 54 + 10 years. Cancer diagnoses were breast (82%) and colorectal (18%). Statistically significant differences (p < 0.05) indicated improvement in the intervention group for: (a) functional well-being, from the FACIT, (Control: M = −0.69, SE = 0.91; Intervention: M = 3.04, SE = 1.13); and (b) self-efficacy in maintaining social relationships, (Control: M = −0.333, SE = 0.33; Intervention: M = 0.621, SE = 0.27). No significant differences were found in health needs, other subscales of quality of life, the extent and number of strategies used in coping and adjusting to cancer and in other domains of self-efficacy. Conclusions While the results should be interpreted with caution, due to the non-randomised nature of the study and the small sample size, they indicate the potential benefits of tailored self-management interventions warrant further investigation in this context.

Review: "Context : A framework for its Influence on Evaluation Practice (New Directions for Evaluation No. 135)" edited by Rog, D. J.; Fitzpatrick, J. L. and Conner, R. F.

This book offers a framework for the influence of context on evaluation practice and is applied to three case studies: environmental context; indigenous context and political context; and finishes with a process for implementation.

Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

REFRAME: a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Kaleidoscope : illuminating learning and teaching evaluation for a shared stakeholder lens

Within the Australian higher education sector, institutions are required to evaluate teaching, units and courses to assure the quality of the student learning experience, however with hardly any regulatory parameters guiding institutions, and with disparate practices, there are few opportunities to benchmakr across institutions or the sector. QUT has received interest and requests from national and international universities on accessing Reframe: QUT's Evaluation Framework.

Evaluation of the alkaline comet assay and urinary 3-methyladenine excretion for monitoring DNA damage in melanoma patients treated with dacarbazine and tamoxifen

Purpose: To develop, using dacarbazine as a model, reliable techniques for measuring DNA damage and repair as pharmacodynamic endpoints for patients receiving chemotherapy. Methods: A group of 39 patients with malignant melanoma were treated with dacarbazine 1 g/m2 i.v. every 21 days. Tamoxifen 20 mg daily was commenced 24 h after the first infusion and continued until 3 weeks after the last cycle of chemotherapy. DNA strand breaks formed during dacarbazine-induced DNA damage and repair were measured in individual cells by the alkaline comet assay. DNA methyl adducts were quantified by measuring urinary 3-methyladenine (3-MeA) excretion using immunoaffinity ELISA. Venous blood was taken on cycles 1 and 2 for separation of peripheral blood lymphocytes (PBLs) for measurement of DNA strand breaks. Results: Wide interpatient variation in PBL DNA strand breaks occurred following chemotherapy, with a peak at 4 h (median 26.6 h, interquartile range 14.75- 40.5 h) and incomplete repair by 24 h. Similarly, there was a range of 3-MeA excretion with peak levels 4-10 h after chemotherapy (median 33 nmol/h, interquartile range 20.448.65 nmol/h). Peak 3-MeA excretion was positively correlated with DNA strand breaks at 4 h (Spearman's correlation coefficient, r = 0.39, P = 0.036) and 24 h (r = 0.46, P = 0.01). Drug-induced emesis correlated with PBL DNA strand breaks (Mann Whitney U-test, P = 0.03) but not with peak 3-MeA excretion. Conclusions: DNA damage and repair following cytotoxic chemotherapy can be measured in vivo by the alkaline comet assay and by urinary 3-MeA excretion in patients receiving chemotherapy.

An evaluation of dog-assisted therapy for residents of aged care facilities with dementia.

Although some research suggests that dog-assisted therapy may be beneficial for people with dementia living in residential aged care facilities, the intervention has not been adequately investigated. To address this shortcoming, we conducted a randomized controlled trial of dog-assisted therapy versus a human-therapist-only intervention for this population. Fifty-five residents with mild to moderate dementia living in three Australian residential aged care facilities completed an 11-week trial of the interventions. Allocation to the intervention was random and participants completed validated measures of mood, psychosocial functioning, and quality of life (QOL), both prior to and following the intervention. No adverse events were associated with the dog-assisted intervention, and following it participants who had worse baseline depression scores demonstrated significantly improved depression scores relative to participants in the human-therapist-only intervention. Participants in the dogassisted intervention also showed significant improvements on a measure of QOL in one facility compared with those in the human-therapist-only group (although worse in another facility that had been affected by an outbreak of gastroenteritis). This study provides some evidence that dog-assisted therapy may be beneficial for some residents of aged care facilities with dementia.

A brief staff dementia education program : implementation and evaluation

Brief dementia education training programs appear to be effective in improving knowledge about dementia and self-confidence in interacting with patients with dementia. It is recommended that brief dementia training sessions be provided on a regular, on-going basis, particularly in view of frequent staff changes in the acute hospital environment.

An evaluation of tumor oxygenation and gene expression in patients with early stage non-small cell lung cancers

Background: To directly assess tumor oxygenation in resectable non - small cell lung cancers (NSCLC) and to correlate tumor pO2 and the selected gene and protein expression to treatment outcomes. Methods: Twenty patients with resectable NSCLC were enrolled. Intraoperative measurements of normal lung and tumor pO2 were done with the Eppendorf polarographic electrode. All patients had plasma osteopontin measurements by ELISA. Carbonic anhydrase-IX (CA IX) staining of tumor sections was done in the majority of patients (n = 16), as was gene expression profiling (n = 12) using cDNA microarrays. Tumor pO2 was correlated with CA IX staining, osteopontin levels, and treatment outcomes. Results: The median tumor pO2 ranged from 0.7 to 46 mm Hg (median, 16.6) and was lower than normal lung pO2 in all but one patient. Because both variables were affected by the completeness of lung deflation during measurement, we used the ratio of tumor/normal lung (T/L) pO2 as a reflection of tumor oxygenation. The median T/L pO 2 was 0.13. T/L pO2 correlated significantly with plasma osteopontin levels (r = 0.53, P = 0.02) and CA IX expression (P = 0.006). Gene expression profiling showed that high CD44 expression was a predictor for relapse, which was confirmed by tissue staining of CD44 variant 6 protein. Other variables associated with the risk of relapse were T stage (P = 0.02), T/L pO2 (P = 0.04), and osteopontin levels (P = 0.001). Conclusions: Tumor hypoxia exists in resectable NSCLC and is associated with elevated expression of osteopontin and CA IX. Tumor hypoxia and elevated osteopontin levels and CD44 expression correlated with poor prognosis. A larger study is needed to confirm the prognostic significance of these factors. © 2006 American Association for Cancer Research.

Evaluation of canine bone marrow-derived mesenchymal stem cells after long-term cryopreservation

Autologous bone marrow-derived mesenchymal stem cell (BMSCs)-based therapies show great potential in regenerative medicine. However, long-term storage and preservation of BMSCs for clinical use is still a great clinical challenge. The present study aimed to analyze the effect of long-term cryopreservation on the regenerative ability of BMSCs. After cryopreservation of BMSCs from beagle dogs for three years, cell viability, and quantitative analysis of alkaline phosphatase (ALP) activity, surface adherence, and mineralized nodule formation were analyzed. BMSCs in cell-scaffold complex were then implanted into nude mice. There was no significant difference in cell viability and ALP activity between osteogenic differentiation and non-osteogenic differentiation of BMSCs, and BMSCs in cell-scaffold complex retained osteogenic differentiation ability in vivo. These results indicate that long-term cryopreserved BMSCs maintain their have capacity to contribute to regeneration.

Learner-focused evaluation cycles : facilitating learning using feedforward, concurrent and feedback evaluation

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors.

Design, implementation and multisite evaluation of a system suitability protocol for the quantitative assessment of instrument performance in liquid chromatography-multiple reaction monitoring-MS (LC-MRM-MS)

Multiple reaction monitoring (MRM) mass spectrometry coupled with stable isotope dilution (SID) and liquid chromatography (LC) is increasingly used in biological and clinical studies for precise and reproducible quantification of peptides and proteins in complex sample matrices. Robust LC-SID-MRM-MS-based assays that can be replicated across laboratories and ultimately in clinical laboratory settings require standardized protocols to demonstrate that the analysis platforms are performing adequately. We developed a system suitability protocol (SSP), which employs a predigested mixture of six proteins, to facilitate performance evaluation of LC-SID-MRM-MS instrument platforms, configured with nanoflow-LC systems interfaced to triple quadrupole mass spectrometers. The SSP was designed for use with low multiplex analyses as well as high multiplex approaches when software-driven scheduling of data acquisition is required. Performance was assessed by monitoring of a range of chromatographic and mass spectrometric metrics including peak width, chromatographic resolution, peak capacity, and the variability in peak area and analyte retention time (RT) stability. The SSP, which was evaluated in 11 laboratories on a total of 15 different instruments, enabled early diagnoses of LC and MS anomalies that indicated suboptimal LC-MRM-MS performance. The observed range in variation of each of the metrics scrutinized serves to define the criteria for optimized LC-SID-MRM-MS platforms for routine use, with pass/fail criteria for system suitability performance measures defined as peak area coefficient of variation <0.15, peak width coefficient of variation <0.15, standard deviation of RT <0.15 min (9 s), and the RT drift <0.5min (30 s). The deleterious effect of a marginally performing LC-SID-MRM-MS system on the limit of quantification (LOQ) in targeted quantitative assays illustrates the use and need for a SSP to establish robust and reliable system performance. Use of a SSP helps to ensure that analyte quantification measurements can be replicated with good precision within and across multiple laboratories and should facilitate more widespread use of MRM-MS technology by the basic biomedical and clinical laboratory research communities.