Measurement as legitimacy versus legitimacy of measures : Performance evaluation of social enterprise

Purpose The purpose of this paper is to review the growing emphasis on quantifiable performance measures such as social return on investment (SROI) in third sector organisations – specifically, social enterprise – through a legitimacy theory lens. It then examines what social enterprises value (i.e. consider important) in terms of performance evaluation, using a case study approach. Design/methodology/approach Case studies involving interviews, documentary analysis, and observation, of three social enterprises at different life-cycle stages with different funding structures, were constructed to consider “what measures matter” from a practitioner's perspective. Findings Findings highlight a priority on quality outcomes and impacts in primarily qualitative terms to evaluate performance. Further, there is a noticeable lack of emphasis on financial measures other than basic access to financial resources to continue pursuing social goals. Social implications The practical challenges faced by social enterprises – many of which are small to medium sized – in evaluating performance and by implication organisational legitimacy are contrasted with measures such as SROI which are resource intensive and have inherent methodological limitations. Hence, findings suggest the limited and valuable resources of social enterprises would be better allocated towards documenting the actual outcomes and impacts as a first step, in order to evaluate social and financial performance in terms appropriate to each objective, in order to demonstrate organisational legitimacy. Originality/value Findings distinguish between processes which may hold symbolic legitimacy for select stakeholder groups, and processes which hold substantive, cognitive legitimacy for stakeholders more broadly, in the under-researched context of social enterprise.

Development and formative evaluation of a motorcycle rider training intervention to address risk taking

The need to address on-road motorcycle safety in Australia is important due to the disproportionately high percentage of riders and pillions killed and injured each year. One approach to preventing motorcycle-related injury is through training and education. However, motorcycle rider training lacks empirical support as an effective road safety countermeasure to reduce crash involvement. Previous reviews have highlighted that risk-taking is a contributing factor in many motorcycle crashes, rather than merely a lack of vehicle-control skills (Haworth & Mulvihill, 2005; Jonah, Dawson & Bragg, 1982; Watson et al, 1996). Hence, though the basic vehicle-handling skills and knowledge of road rules that are taught in most traditional motorcycle licence training programs may be seen as an essential condition of safe riding, they do not appear to be sufficient in terms of crash reduction. With this in mind there is considerable scope for the improvement of program focus and content for rider training and education. This program of research examined an existing traditional pre-licence motorcycle rider training program and formatively evaluated the addition of a new classroom-based module to address risky riding; the Three Steps to Safer Riding program. The pilot program was delivered in the real world context of the Q-Ride motorcycle licensing system in the state of Queensland, Australia. Three studies were conducted as part of the program of research: Study 1, a qualitative investigation of delivery practices and student learning needs in an existing rider training course; Study 2, an investigation of the extent to which an existing motorcycle rider training course addressed risky riding attitudes and motives; and Study 3, a formative evaluation of the new program. A literature review as well as the investigation of learning needs for motorcyclists in Study 1 aimed to inform the initial planning and development of the Three Steps to Safer Riding program. Findings from Study 1 suggested that the training delivery protocols used by the industry partner training organisation were consistent with a learner-centred approach and largely met the learning needs of trainee riders. However, it also found that information from the course needs to be reinforced by on-road experiences for some riders once licensed and that personal meaning for training information was not fully gained until some riding experience had been obtained. While this research informed the planning and development of the new program, a project team of academics and industry experts were responsible for the formulation of the final program. Study 2 and Study 3 were conducted for the purpose of formative evaluation and program refinement. Study 2 served primarily as a trial to test research protocols and data collection methods with the industry partner organisation and, importantly, also served to gather comparison data for the pilot program which was implemented with the same rider training organisation. Findings from Study 2 suggested that the existing training program of the partner organisation generally had a positive (albeit small) effect on safety in terms of influencing attitudes to risk taking, the propensity for thrill seeking, and intentions to engage in future risky riding. However, maintenance of these effects over time and the effects on riding behaviour remain unclear due to a low response rate upon follow-up 24 months after licensing. Study 3 was a formative evaluation of the new pilot program to establish program effects and possible areas for improvement. Study 3a examined the short term effects of the intervention pilot on psychosocial factors underpinning risky riding compared to the effects of the standard traditional training program (examined in Study 2). It showed that the course which included the Three Steps to Safer Riding program elicited significantly greater positive attitude change towards road safety than the existing standard licensing course. This effect was found immediately following training, and mean scores for attitudes towards safety were also maintained at the 12 month follow-up. The pilot program also had an immediate effect on other key variables such as risky riding intentions and the propensity for thrill seeking, although not significantly greater than the traditional standard training. A low response rate at the 12 month follow-up unfortunately prevented any firm conclusions being drawn regarding the impact of the pilot program on self-reported risky riding once licensed. Study 3a further showed that the use of intermediate outcomes such as self-reported attitudes and intentions for evaluation purposes provides insights into the mechanisms underpinning risky riding that can be changed by education and training. A multifaceted process evaluation conducted in Study 3b confirmed that the intervention pilot was largely delivered as designed, with course participants also rating most aspects of training delivery highly. The complete program of research contributed to the overall body of knowledge relating to motorcycle rider training, with some potential implications for policy in the area of motorcycle rider licensing. A key finding of the research was that psychosocial influences on risky riding can be shaped by structured education that focuses on awareness raising at a personal level and provides strategies to manage future riding situations. However, the formative evaluation was mainly designed to identify areas of improvement for the Three Steps to Safer Riding program and found several areas of potential refinement to improve future efficacy of the program. This included aspects of program content, program delivery, resource development, and measurement tools. The planned future follow-up of program participants' official crash and traffic offence records over time may lend further support for the application of the program within licensing systems. The findings reported in this thesis offer an initial indication that the Three Steps to Safer Riding is a useful resource to accompany skills-based training programs.

Higher harmonic large-amplitude fourier transformed alternating current voltammetry : analytical attributes derived from studies of the oxidation of ferrocenemethanol and uric acid at a glassy carbon electrode

An analytical evaluation of the higher ac harmonic components derived from large amplitude Fourier transformed voltammetry is provided for the reversible oxidation of ferrocenemethanol (FcMeOH) and oxidation of uric acid by an EEC mechanism in a pH 7.4 phosphate buffer at a glassy carbon (GC) electrode. The small background current in the analytically optimal fifth harmonic is predominantly attributed to faradaic current associated with the presence of electroactive functional groups on the GC electrode surface, rather than to capacitive current which dominates the background in the dc, and the initial three ac harmonics. The detection limits for the dc and the first to fifth harmonic ac components are 1.9, 5.89, 2.1, 2.5, 0.8, and 0.5 µM for FcMeOH, respectively, using a sine wave modulation of 100 mV at 21.46 Hz and a dc sweep rate of 111.76 mV s−1. Analytical performance then progressively deteriorates in the sixth and higher harmonics. For the determination of uric acid, the capacitive background current was enhanced and the reproducibility lowered by the presence of surface active uric acid, but the rapid overall 2e− rather than 1e– electron transfer process gives rise to a significantly enhanced fifth harmonic faradaic current which enabled a detection limit of 0.3 µM to be achieved which is similar to that reported using chemically modified electrodes. Resolution of overlapping voltammetric signals for a mixture of uric acid and dopamine is also achieved using higher fourth or fifth harmonic components, under very low background current conditions. The use of higher fourth and fifth harmonics exhibiting highly favorable faradaic to background (noise) current ratios should therefore be considered in analytical applications under circumstances where the electron transfer rate is fast.

Evaluation of an ambulatory nutrition support service for malnourished patients post hospital discharge

Introduction Malnutrition is common among hospitalised patients, with poor follow-up of nutrition support post-discharge. Published studies on the efficacy of ambulatory nutrition support (ANS) for malnourished patients post-discharge are scarce. The aims of this study were to evaluate the rate of dietetics follow-up of malnourished patients post-discharge, before (2008) and after (2010) implementation of a new ANS service, and to evaluate nutritional outcomes post-implementation. Materials and Methods Consecutive samples of 261 (2008) and 163 (2010) adult inpatients referred to dietetics and assessed as malnourished using Subjective Global Assessment (SGA) were enrolled. All subjects received inpatient nutrition intervention and dietetic outpatient clinic follow-up appointments. For the 2010 cohort, ANS was initiated to provide telephone follow-up and home visits for patients who failed to attend the outpatient clinic. Subjective Global Assessment, body weight, quality of life (EQ-5D VAS) and handgrip strength were measured at baseline and five months post-discharge. Paired t-test was used to compare pre- and post-intervention results. Results In 2008, only 15% of patients returned for follow-up with a dietitian within four months post-discharge. After implementation of ANS in 2010, the follow-up rate was 100%. Mean weight improved from 44.0 ± 8.5kg to 46.3 ± 9.6kg, EQ-5D VAS from 61.2 ± 19.8 to 71.6 ± 17.4 and handgrip strength from 15.1 ± 7.1 kg force to 17.5 ± 8.5 kg force; p<0.001 for all. Seventy-four percent of patients improved in SGA score. Conclusion Ambulatory nutrition support resulted in significant improvements in follow-up rate, nutritional status and quality of life of malnourished patients post-discharge.

Review: "Context : A framework for its Influence on Evaluation Practice (New Directions for Evaluation No. 135)" edited by Rog, D. J.; Fitzpatrick, J. L. and Conner, R. F.

This book offers a framework for the influence of context on evaluation practice and is applied to three case studies: environmental context; indigenous context and political context; and finishes with a process for implementation.

Incorporating translation in qualitative studies : two case studies in education

Cross-language qualitative research in education continues to increase. However, there has been inadequate discussion in the literature concerning the translation process that ensures research trustworthiness applicable for bilingual researchers. Informed by the literature on evaluation criteria for qualitative data translation, this paper compares two different procedures for incorporating translation in education qualitative research to provide a clear depiction of the complexities involved in translating qualitative data and the strengths and weaknesses of each procedure. To maintain the trustworthiness of the qualitative research, it is necessary to minimise translation errors, provide detailed accounts of the translation process, involve more than one translator and remain open to scrutiny from those seeking to access the translation process. Taking into account the resource constraints often faced by novice qualitative researchers, this paper provides some strategies that can be employed in similar contexts.

Learner-focused evaluation cycles : facilitating learning using feedforward, concurrent and feedback evaluation

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors.

Design, implementation and multisite evaluation of a system suitability protocol for the quantitative assessment of instrument performance in liquid chromatography-multiple reaction monitoring-MS (LC-MRM-MS)

Multiple reaction monitoring (MRM) mass spectrometry coupled with stable isotope dilution (SID) and liquid chromatography (LC) is increasingly used in biological and clinical studies for precise and reproducible quantification of peptides and proteins in complex sample matrices. Robust LC-SID-MRM-MS-based assays that can be replicated across laboratories and ultimately in clinical laboratory settings require standardized protocols to demonstrate that the analysis platforms are performing adequately. We developed a system suitability protocol (SSP), which employs a predigested mixture of six proteins, to facilitate performance evaluation of LC-SID-MRM-MS instrument platforms, configured with nanoflow-LC systems interfaced to triple quadrupole mass spectrometers. The SSP was designed for use with low multiplex analyses as well as high multiplex approaches when software-driven scheduling of data acquisition is required. Performance was assessed by monitoring of a range of chromatographic and mass spectrometric metrics including peak width, chromatographic resolution, peak capacity, and the variability in peak area and analyte retention time (RT) stability. The SSP, which was evaluated in 11 laboratories on a total of 15 different instruments, enabled early diagnoses of LC and MS anomalies that indicated suboptimal LC-MRM-MS performance. The observed range in variation of each of the metrics scrutinized serves to define the criteria for optimized LC-SID-MRM-MS platforms for routine use, with pass/fail criteria for system suitability performance measures defined as peak area coefficient of variation <0.15, peak width coefficient of variation <0.15, standard deviation of RT <0.15 min (9 s), and the RT drift <0.5min (30 s). The deleterious effect of a marginally performing LC-SID-MRM-MS system on the limit of quantification (LOQ) in targeted quantitative assays illustrates the use and need for a SSP to establish robust and reliable system performance. Use of a SSP helps to ensure that analyte quantification measurements can be replicated with good precision within and across multiple laboratories and should facilitate more widespread use of MRM-MS technology by the basic biomedical and clinical laboratory research communities.

Percutaneous coronary intervention with drug-eluting stents or coronary artery bypass surgery in subjects with type 2 diabetes : evaluation of: Farkouth ME, Domanski M, Sleeper LA, et al. Strategies for multivessel revascularization in patients with diabetes. N Engl J Med 2012;367(25):2375-84, and Contini GA, Nicolini F, Fortuna D, et al. Five-year outcomes of surgical or percutaneous myocardial revascularization in diabetic patients. Int J Cardiol 2012. [Epub ahead of print]

There is debate as to whether percutaneous coronary intervention (PCI) with drug-eluting stents or coronary artery bypass surgery (CABG) is the best procedure for subjects with type 2 diabetes and coronary artery disease requiring revascularization. There is some evidence that following these procedures there is less further revascularization with CABG than PCI in subjects with diabetes. Two recent studies; the FREEDOM (Future Revascularization Evaluation in patients with Diabetes mellitus: Optimal Management of Multivessel Disease) trial, and a trial using a real world diabetic population from a Registry, have shown that the benefits of CABG over PCI in subjects with type 2 diabetes extends to lower rates of death and myocardial infarct, in addition to lower rates of revascularization. However, the rates of stroke may be higher with CABG than PCI with drug-eluting stents in this population. Thus, if CABG is going to be preferred to PCI in subjects with type 2 diabetes and multivessel coronary disease, consideration should be given to how to reduce the rates of stroke with CABG.

Facilitating local stories in post-disaster regional communities : evaluation in narrative-driven oral history projects

Cyclone Yasi struck the Cassowary Coast of Northern Queensland, Australia, in the early hours of February 3, 2011, destroying many homes and property, including the destruction of the Cardwell and district historical society’s premises. With their own homes flattened, many residents were forced to live in mobile accommodation, with extended family, or leave the area altogether. The historical society members seemed, however, particularly devastated by their flattened foreshore museum and loss of their precious collection of material. A call for assistance was made through the Oral History Association of Australia’s Queensland branch (OHAA Qld), which along with a Queensland University of Technology (QUT) research team sponsored a trip to best plan how they could start to pick up the pieces to rebuild the museum. This chapter highlights the need for communities to gather, preserve and present their own stories, in a way that is sustainable and meaningful to them – whether that be because of a disaster, or as they go about life in their contemporary communities – the key being that good advice, professional support and embedded evaluation practices at crucial moments along the way can be critically important.

Academic legal research and academic research capacity enhancement working topic : "purposes and objectives of law schools beyond educating students (research, capacity building,community service and outreach)"

One of the main objectives of law schools beyond educating students is to produce viable legal research. The comments in this paper are basically confined to the Australian context, and to examine this topic effectively, it is necessary to briefly review the current tertiary research agenda in Australia. This paper argues that there is a need for recognition and support for an expanded legal research framework along with additional research training for legal academics. There also needs to be more effective methods of measuring and recognising quality in legal research. This method needs to be one that can engender respect in an interdisciplinary context.

An evaluation of the Australian ‘reconceptualising early mathematics learning’ project : key findings and implications

The Pattern and Structure Mathematics Awareness Project (PASMAP) has investigated the development of patterning and early algebraic reasoning among 4 to 8 year olds over a series of related studies. We assert that an awareness of mathematical pattern and structure (AMPS) enables mathematical thinking and simple forms of generalization from an early age. This paper provides an overview of key findings of the Reconceptualizing Early Mathematics Learning empirical evaluation study involving 316 Kindergarten students from 4 schools. The study found highly significant differences on PASA scores for PASMAP students. Analysis of structural development showed increased levels for the PASMAP students; those categorised as low ability developed improved structural responses over a short period of time.

Diabetes mellitus in South Asia : scientific evaluation of the research output

BACKGROUND: Diabetes in South Asia represents a different disease entity in terms of its onset, progression, and complications. In the present study, we systematically analyzed the medical research output on diabetes in South Asia. METHODS: The online SciVerse Scopus database was searched using the search terms "diabetes" and "diabetes mellitus" in the article Title, Abstract or Keywords fields, in conjunction with the names of each regional country in the Author Affiliation field. RESULTS: In total, 8478 research articles were identified. Most were from India (85.1%) and Pakistan (9.6%) and the contribution to the global diabetes research output was 2.1%. Publications from South Asia increased markedly after 2007, with 58.7% of papers published between 2000 and 2010 being published after 2007. Most papers were Research Articles (75.9%) and Reviews (12.9%), with only 90 (1.1%) clinical trials. Publications predominantly appeared in local national journals. Indian authors and institutions had the most number of articles and the highest h-index. There were 136 (1.6%) intraregional collaborative studies. Only 39 articles (0.46%) had >100 citations. CONCLUSIONS: Regional research output on diabetes mellitus is unsatisfactory, with only a minimal contribution to global diabetes research. Publications are not highly cited and only a few randomized controlled trials have been performed. In the coming decades, scientists in the region must collaborate and focus on practical and culturally acceptable interventional studies on diabetes mellitus.

Revisiting sub-topic retrieval in the ImageCLEF 2009 photo retrieval task

Ranking documents according to the Probability Ranking Principle has been theoretically shown to guarantee optimal retrieval effectiveness in tasks such as ad hoc document retrieval. This ranking strategy assumes independence among document relevance assessments. This assumption, however, often does not hold, for example in the scenarios where redundancy in retrieved documents is of major concern, as it is the case in the sub–topic retrieval task. In this chapter, we propose a new ranking strategy for sub–topic retrieval that builds upon the interdependent document relevance and topic–oriented models. With respect to the topic– oriented model, we investigate both static and dynamic clustering techniques, aiming to group topically similar documents. Evidence from clusters is then combined with information about document dependencies to form a new document ranking. We compare and contrast the proposed method against state–of–the–art approaches, such as Maximal Marginal Relevance, Portfolio Theory for Information Retrieval, and standard cluster–based diversification strategies. The empirical investigation is performed on the ImageCLEF 2009 Photo Retrieval collection, where images are assessed with respect to sub–topics of a more general query topic. The experimental results show that our approaches outperform the state–of–the–art strategies with respect to a number of diversity measures.

Evaluation and social impact measurement amongst small to medium social enterprises : process, purpose and value

In recent years, the imperative to communicate organisational impacts to a variety of stakeholders has gained increasing importance within all sectors. Despite growing external demands for evaluation and social impact measurement, there has been limited critically informed analysis about the presumed importance of these activities to organisational success and the practical challenges faced by organisations in undertaking such assessment. In this paper, we present the findings from an action research study of five Australian small to medium social enterprises’ practices and use of evaluation and social impact analysis. Our findings have implications for social enterprise operators, policy makers and social investors regarding when, why and at what level these activities contribute to organisational performance and the fulfilment of mission.