Review: "Context : A framework for its Influence on Evaluation Practice (New Directions for Evaluation No. 135)" edited by Rog, D. J.; Fitzpatrick, J. L. and Conner, R. F.

This book offers a framework for the influence of context on evaluation practice and is applied to three case studies: environmental context; indigenous context and political context; and finishes with a process for implementation.

Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

REFRAME: a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Kaleidoscope : illuminating learning and teaching evaluation for a shared stakeholder lens

Within the Australian higher education sector, institutions are required to evaluate teaching, units and courses to assure the quality of the student learning experience, however with hardly any regulatory parameters guiding institutions, and with disparate practices, there are few opportunities to benchmakr across institutions or the sector. QUT has received interest and requests from national and international universities on accessing Reframe: QUT's Evaluation Framework.

Evaluation of the alkaline comet assay and urinary 3-methyladenine excretion for monitoring DNA damage in melanoma patients treated with dacarbazine and tamoxifen

Purpose: To develop, using dacarbazine as a model, reliable techniques for measuring DNA damage and repair as pharmacodynamic endpoints for patients receiving chemotherapy. Methods: A group of 39 patients with malignant melanoma were treated with dacarbazine 1 g/m2 i.v. every 21 days. Tamoxifen 20 mg daily was commenced 24 h after the first infusion and continued until 3 weeks after the last cycle of chemotherapy. DNA strand breaks formed during dacarbazine-induced DNA damage and repair were measured in individual cells by the alkaline comet assay. DNA methyl adducts were quantified by measuring urinary 3-methyladenine (3-MeA) excretion using immunoaffinity ELISA. Venous blood was taken on cycles 1 and 2 for separation of peripheral blood lymphocytes (PBLs) for measurement of DNA strand breaks. Results: Wide interpatient variation in PBL DNA strand breaks occurred following chemotherapy, with a peak at 4 h (median 26.6 h, interquartile range 14.75- 40.5 h) and incomplete repair by 24 h. Similarly, there was a range of 3-MeA excretion with peak levels 4-10 h after chemotherapy (median 33 nmol/h, interquartile range 20.448.65 nmol/h). Peak 3-MeA excretion was positively correlated with DNA strand breaks at 4 h (Spearman's correlation coefficient, r = 0.39, P = 0.036) and 24 h (r = 0.46, P = 0.01). Drug-induced emesis correlated with PBL DNA strand breaks (Mann Whitney U-test, P = 0.03) but not with peak 3-MeA excretion. Conclusions: DNA damage and repair following cytotoxic chemotherapy can be measured in vivo by the alkaline comet assay and by urinary 3-MeA excretion in patients receiving chemotherapy.

An evaluation of dog-assisted therapy for residents of aged care facilities with dementia.

Although some research suggests that dog-assisted therapy may be beneficial for people with dementia living in residential aged care facilities, the intervention has not been adequately investigated. To address this shortcoming, we conducted a randomized controlled trial of dog-assisted therapy versus a human-therapist-only intervention for this population. Fifty-five residents with mild to moderate dementia living in three Australian residential aged care facilities completed an 11-week trial of the interventions. Allocation to the intervention was random and participants completed validated measures of mood, psychosocial functioning, and quality of life (QOL), both prior to and following the intervention. No adverse events were associated with the dog-assisted intervention, and following it participants who had worse baseline depression scores demonstrated significantly improved depression scores relative to participants in the human-therapist-only intervention. Participants in the dogassisted intervention also showed significant improvements on a measure of QOL in one facility compared with those in the human-therapist-only group (although worse in another facility that had been affected by an outbreak of gastroenteritis). This study provides some evidence that dog-assisted therapy may be beneficial for some residents of aged care facilities with dementia.

A brief staff dementia education program : implementation and evaluation

Brief dementia education training programs appear to be effective in improving knowledge about dementia and self-confidence in interacting with patients with dementia. It is recommended that brief dementia training sessions be provided on a regular, on-going basis, particularly in view of frequent staff changes in the acute hospital environment.

An evaluation of tumor oxygenation and gene expression in patients with early stage non-small cell lung cancers

Background: To directly assess tumor oxygenation in resectable non - small cell lung cancers (NSCLC) and to correlate tumor pO2 and the selected gene and protein expression to treatment outcomes. Methods: Twenty patients with resectable NSCLC were enrolled. Intraoperative measurements of normal lung and tumor pO2 were done with the Eppendorf polarographic electrode. All patients had plasma osteopontin measurements by ELISA. Carbonic anhydrase-IX (CA IX) staining of tumor sections was done in the majority of patients (n = 16), as was gene expression profiling (n = 12) using cDNA microarrays. Tumor pO2 was correlated with CA IX staining, osteopontin levels, and treatment outcomes. Results: The median tumor pO2 ranged from 0.7 to 46 mm Hg (median, 16.6) and was lower than normal lung pO2 in all but one patient. Because both variables were affected by the completeness of lung deflation during measurement, we used the ratio of tumor/normal lung (T/L) pO2 as a reflection of tumor oxygenation. The median T/L pO 2 was 0.13. T/L pO2 correlated significantly with plasma osteopontin levels (r = 0.53, P = 0.02) and CA IX expression (P = 0.006). Gene expression profiling showed that high CD44 expression was a predictor for relapse, which was confirmed by tissue staining of CD44 variant 6 protein. Other variables associated with the risk of relapse were T stage (P = 0.02), T/L pO2 (P = 0.04), and osteopontin levels (P = 0.001). Conclusions: Tumor hypoxia exists in resectable NSCLC and is associated with elevated expression of osteopontin and CA IX. Tumor hypoxia and elevated osteopontin levels and CD44 expression correlated with poor prognosis. A larger study is needed to confirm the prognostic significance of these factors. © 2006 American Association for Cancer Research.

Evaluation of canine bone marrow-derived mesenchymal stem cells after long-term cryopreservation

Autologous bone marrow-derived mesenchymal stem cell (BMSCs)-based therapies show great potential in regenerative medicine. However, long-term storage and preservation of BMSCs for clinical use is still a great clinical challenge. The present study aimed to analyze the effect of long-term cryopreservation on the regenerative ability of BMSCs. After cryopreservation of BMSCs from beagle dogs for three years, cell viability, and quantitative analysis of alkaline phosphatase (ALP) activity, surface adherence, and mineralized nodule formation were analyzed. BMSCs in cell-scaffold complex were then implanted into nude mice. There was no significant difference in cell viability and ALP activity between osteogenic differentiation and non-osteogenic differentiation of BMSCs, and BMSCs in cell-scaffold complex retained osteogenic differentiation ability in vivo. These results indicate that long-term cryopreserved BMSCs maintain their have capacity to contribute to regeneration.

Learner-focused evaluation cycles : facilitating learning using feedforward, concurrent and feedback evaluation

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors.

Percutaneous coronary intervention with drug-eluting stents or coronary artery bypass surgery in subjects with type 2 diabetes : evaluation of: Farkouth ME, Domanski M, Sleeper LA, et al. Strategies for multivessel revascularization in patients with diabetes. N Engl J Med 2012;367(25):2375-84, and Contini GA, Nicolini F, Fortuna D, et al. Five-year outcomes of surgical or percutaneous myocardial revascularization in diabetic patients. Int J Cardiol 2012. [Epub ahead of print]

There is debate as to whether percutaneous coronary intervention (PCI) with drug-eluting stents or coronary artery bypass surgery (CABG) is the best procedure for subjects with type 2 diabetes and coronary artery disease requiring revascularization. There is some evidence that following these procedures there is less further revascularization with CABG than PCI in subjects with diabetes. Two recent studies; the FREEDOM (Future Revascularization Evaluation in patients with Diabetes mellitus: Optimal Management of Multivessel Disease) trial, and a trial using a real world diabetic population from a Registry, have shown that the benefits of CABG over PCI in subjects with type 2 diabetes extends to lower rates of death and myocardial infarct, in addition to lower rates of revascularization. However, the rates of stroke may be higher with CABG than PCI with drug-eluting stents in this population. Thus, if CABG is going to be preferred to PCI in subjects with type 2 diabetes and multivessel coronary disease, consideration should be given to how to reduce the rates of stroke with CABG.

Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

Evaluation of brief dietary questions to estimate vegetable and fruit consumption – using serum carotenoids and red-cell folate

Objective: To evaluate responses to self-administered brief questions regarding consumption of vegetables and fruit by comparison with blood levels of serum carotenoids and red-cell folate. Design: A cross-sectional study in which participants reported their usual intake of fruit and vegetables in servings per day, and serum levels of five carotenoids (α-carotene, β-carotene, β-cryptoxanthin, lutein/zeaxanthin and lycopene) and red-cell folate were measured. Serum carotenoid levels were determined by high-performance liquid chromatography, and red-cell folate by an automated immunoassay system. Settings and subjects: Between October and December 2000, a sample of 1598 adults aged 25 years and over, from six randomly selected urban centres in Queensland, Australia, were examined as part of a national study conducted to determine the prevalence of diabetes and associated cardiovascular risk factors. Results: Statistically significant (P<0.01) associations with vegetable and fruit intake (categorised into groups: ≤1 serving, 2–3 servings and ≥4 servings per day) were observed for α-carotene, β-carotene, β-cryptoxanthin, lutein/zeaxanthin and red-cell folate. The mean level of these carotenoids and of red-cell folate increased with increasing frequency of reported servings of vegetables and fruit, both before and after adjusting for potential confounding factors. A significant association with lycopene was observed only for vegetable intake before adjusting for confounders. Conclusions: These data indicate that brief questions may be a simple and valuable tool for monitoring vegetable and fruit intake in this population.

An evaluation of different features and learning models for anomalous event detection

The huge amount of CCTV footage available makes it very burdensome to process these videos manually through human operators. This has made automated processing of video footage through computer vision technologies necessary. During the past several years, there has been a large effort to detect abnormal activities through computer vision techniques. Typically, the problem is formulated as a novelty detection task where the system is trained on normal data and is required to detect events which do not fit the learned ‘normal’ model. There is no precise and exact definition for an abnormal activity; it is dependent on the context of the scene. Hence there is a requirement for different feature sets to detect different kinds of abnormal activities. In this work we evaluate the performance of different state of the art features to detect the presence of the abnormal objects in the scene. These include optical flow vectors to detect motion related anomalies, textures of optical flow and image textures to detect the presence of abnormal objects. These extracted features in different combinations are modeled using different state of the art models such as Gaussian mixture model(GMM) and Semi- 2D Hidden Markov model(HMM) to analyse the performances. Further we apply perspective normalization to the extracted features to compensate for perspective distortion due to the distance between the camera and objects of consideration. The proposed approach is evaluated using the publicly available UCSD datasets and we demonstrate improved performance compared to other state of the art methods.

A methodology for effective implementation of lean strategies and its performance evaluation in manufacturing organizations

Purpose – The purpose of this paper is to develop an effective methodology for implementing lean manufacturing strategies and a leanness evaluation metric using continuous performance measurement (CPM). Design/methodology/approach – Based on five lean principles, a systematic lean implementation methodology for manufacturing organizations has been proposed. A simplified leanness evaluation metric consisting of both efficiency and effectiveness attributes of manufacturing performance has been developed for continuous evaluation of lean implementation. A case study to validate the proposed methodology has been conducted and proposed CPM metric has been used to assess the manufacturing leanness. Findings – Proposed methodology is able to systematically identify manufacturing wastes, select appropriate lean tools, identify relevant performance indicators, achieve significant performance improvement and establish lean culture in the organization. Continuous performance measurement matrices in terms of efficiency and effectiveness are proved to be appropriate methods for continuous evaluation of lean performance. Research limitations/implications – Effectiveness of the method developed has been demonstrated by applying it in a real life assembly process. However, more tests/applications will be necessary to generalize the findings. Practical implications – Results show that applying the methods developed, managers can successfully identify and remove manufacturing wastes from their production processes. By improving process efficiency, they can optimize their resource allocations. Manufacturers now have a validated step by step methodology for successfully implementing lean strategies. Originality/value – According to the authors’ best knowledge, this is the first known study that proposed a systematic lean implementation methodology based on lean principles and continuous improvement techniques. Evaluation of performance improvement by lean strategies is a critical issue. This study develops a simplified leanness evaluation metric considering both efficiency and effectiveness attributes and integrates it with the lean implementation methodology.