Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

A methodology for effective implementation of lean strategies and its performance evaluation in manufacturing organizations

Purpose – The purpose of this paper is to develop an effective methodology for implementing lean manufacturing strategies and a leanness evaluation metric using continuous performance measurement (CPM). Design/methodology/approach – Based on five lean principles, a systematic lean implementation methodology for manufacturing organizations has been proposed. A simplified leanness evaluation metric consisting of both efficiency and effectiveness attributes of manufacturing performance has been developed for continuous evaluation of lean implementation. A case study to validate the proposed methodology has been conducted and proposed CPM metric has been used to assess the manufacturing leanness. Findings – Proposed methodology is able to systematically identify manufacturing wastes, select appropriate lean tools, identify relevant performance indicators, achieve significant performance improvement and establish lean culture in the organization. Continuous performance measurement matrices in terms of efficiency and effectiveness are proved to be appropriate methods for continuous evaluation of lean performance. Research limitations/implications – Effectiveness of the method developed has been demonstrated by applying it in a real life assembly process. However, more tests/applications will be necessary to generalize the findings. Practical implications – Results show that applying the methods developed, managers can successfully identify and remove manufacturing wastes from their production processes. By improving process efficiency, they can optimize their resource allocations. Manufacturers now have a validated step by step methodology for successfully implementing lean strategies. Originality/value – According to the authors’ best knowledge, this is the first known study that proposed a systematic lean implementation methodology based on lean principles and continuous improvement techniques. Evaluation of performance improvement by lean strategies is a critical issue. This study develops a simplified leanness evaluation metric considering both efficiency and effectiveness attributes and integrates it with the lean implementation methodology.

Writing the Barbaric Recent Past : Holocaust Fiction for Young Adults

Horror and redemption in Holocaust writing for young adults: Marcus Zusak’s The Book Thief, John Boyne’s The Boy in the Striped Pyjamas. While it has long been thought that the Holocaust is not an appropriate subject matter for young audiences, from The Diary of Anne Frank onwards it has always been part of their reading matter. Never, however, has there been so much interest as in the recent best-selling publications by Zusak and Boyne (the latter of which has been made into a film). This chapter examines the politics of crafting stories for young people about the unspeakable events of the recent past, about who has the right to ‘speak for’ the victims, and whether some genres (for example, fairy stories or fabulism) work best, given the horrific nature of the subject matter.

Evaluation framework for green procurement in road construction

Purpose Most barriers and enablers of sustainable projects are related to procurement. This study proposes a framework for evaluating green procurement practices throughout the lifecycle of road construction projects and demonstrates its application through an Australian case study. Design/methodology/approach The study is based on linking the phases of road construction with incentive mechanisms for proactively motivating behavioural change. A holistic view on utilised and potential incentives is attempted with a literature review and a state-of-practice review. The latter is based on interviews and 90 policy and procurement documents across five Australian states. Findings An evaluation framework with seven procurement stages is suggested to describe current state green procurement incentives throughout the delivery lifecycle of road construction projects. The Australian case study was found to provide useful data to identify gaps and strong points of the different states regarding their level of integration of sustainability and greenhouse gas emissions GHG) reduction elements in their procurement practices. This understanding was used to draw recommendations on future advancement of green procurement. Originality/value: Government entities across the globe can impact considerably the achievement of sustainability and GHG targets, by using their procurement practices and requirements to create incentives for contractors and suppliers to engage in more GHG conscious practices. The present study provides a systematic account of how green procurement practices can be underpinned using the Australian road construction industry as a case study, and distinguish between strong and weak links in the green procurement chain to draw recommendations for future initiatives.

Teacher Evidence Matrix : an evaluation using multiple lines of evidence

Welcome to the Teacher evidence matrix. This matrix is designed for highly qualified discipline experts to evaluate their teaching in a systematic manner. The primary purpose of the Teacher evidence matrix is to provide a tool that an academic staff member at university can annually review their teaching. The annual review will result in you being ready for performance, planning and review; promotion; awards; or employment application. This tool is designed for individual use and will lead to an action plan for implementation.

Visually impaired drivers who use bioptic telescopes : self-assessed driving skills and agreement with on-road driving evaluation

Purpose. To compare self-assessed driving habits and skills of licensed drivers with central visual loss who use bioptic telescopes to those of age-matched normally sighted drivers, and to examine the association between bioptic drivers' impressions of the quality of their driving and ratings by a “backseat” evaluator. Methods. Participants were licensed bioptic drivers (n = 23) and age-matched normally sighted drivers (n = 23). A questionnaire was administered addressing driving difficulty, space, quality, exposure, and, for bioptic drivers, whether the telescope was helpful in on-road situations. Visual acuity and contrast sensitivity were assessed. Information on ocular diagnosis, telescope characteristics, and bioptic driving experience was collected from the medical record or in interview. On-road driving performance in regular traffic conditions was rated independently by two evaluators. Results. Like normally sighted drivers, bioptic drivers reported no or little difficulty in many driving situations (e.g., left turns, rush hour), but reported more difficulty under poor visibility conditions and in unfamiliar areas (P < 0.05). Driving exposure was reduced in bioptic drivers (driving 250 miles per week on average vs. 410 miles per week for normally sighted drivers, P = 0.02), but driving space was similar to that of normally sighted drivers (P = 0.29). All but one bioptic driver used the telescope in at least one driving task, and 56% used the telescope in three or more tasks. Bioptic drivers' judgments about the quality of their driving were very similar to backseat evaluators' ratings. Conclusions. Bioptic drivers show insight into the overall quality of their driving and areas in which they experience driving difficulty. They report using the bioptic telescope while driving, contrary to previous claims that it is primarily used to pass the vision screening test at licensure.

Building indicators in media assistance; four ways evaluation reports present “change”

The international aid and development community has supported programs that aim to build the capacity of media professionals or contribute to an enabling environment throughout the past 20 years. However, two decades on from the first modern media assistance programs, the sector is still struggling to identify, measure and understand the changes effected by their programs. There are questions raised as to whether it is even feasible to identify impacts on society and governance. This paper draws on some preliminary findings from a comparative thematic analysis of 47 evaluation documents of media assistance programs. The aim of this analysis is to identify trends in impact evaluation practice in the media assistance field, as well as the strengths and weaknesses of different evaluation approaches. This paper presents four types of social change claims commonly presented in reports; hypothetical changes, introduction of new opportunities, concrete examples of immediate impacts, and analysis of ongoing social and political changes. Although these types may appear as a spectrum from weak to strong, the interactions are perhaps more accurately understood using metaphors such as building blocks. This paper explores these types in more detail and suggests that a robust set of impacts-types could be useful in developing more grounded theories of change and indicators.

Social event detection at MediaEval 2013 : challenges, datasets and evaluation

In this paper, we provide an overview of the Social Event Detection (SED) task that is part of the MediaEval Bench mark for Multimedia Evaluation 2013. This task requires participants to discover social events and organize the re- lated media items in event-specific clusters within a collection of Web multimedia. Social events are events that are planned by people, attended by people and for which the social multimedia are also captured by people. We describe the challenges, datasets, and the evaluation methodology.

Predicting fault-prone software modules with rank sum classification

The detection and correction of defects remains among the most time consuming and expensive aspects of software development. Extensive automated testing and code inspections may mitigate their effect, but some code fragments are necessarily more likely to be faulty than others, and automated identification of fault prone modules helps to focus testing and inspections, thus limiting wasted effort and potentially improving detection rates. However, software metrics data is often extremely noisy, with enormous imbalances in the size of the positive and negative classes. In this work, we present a new approach to predictive modelling of fault proneness in software modules, introducing a new feature representation to overcome some of these issues. This rank sum representation offers improved or at worst comparable performance to earlier approaches for standard data sets, and readily allows the user to choose an appropriate trade-off between precision and recall to optimise inspection effort to suit different testing environments. The method is evaluated using the NASA Metrics Data Program (MDP) data sets, and performance is compared with existing studies based on the Support Vector Machine (SVM) and Naïve Bayes (NB) Classifiers, and with our own comprehensive evaluation of these methods.

The evaluation of a clinical scar scale for porcine burn scars

This study describes the evaluation of a clinical scar scale for our porcine burn scars, which includes scar cosmetic outcome, colour, height and hair, supplemented with reference porcine scar photographs representing each scar outcome and scar colour scores. A total of 72 porcine burn scars at week 6 after burn were rated in vivo and/or on photographs. Good agreements were achieved for both intra-rater reliability (correlation is 0.86-0.98) and inter-rater reliability (ICC=80-85%). The results showed statistically significant correlations for each pair in this clinical scar scale (p<0.01), with the best correlation found between scar cosmetic outcome and scar colour. A multivariate principle components analysis revealed that this clinical scar assessment was highly correlated with scar histology, wound size, and re-epithelialisation data (p<0.001). More severe scars are clinically characterised by darker purple colouration, more elevation, no presence of hair, histologically by thicker scar tissue, thinner remaining normal dermis, are more likely to have worse contraction, and slower re-epithelialisation. This study demonstrates that our clinical scar scale is a reliable, independent and valuable tool for assessing porcine burn outcome and truthfully reflects scar appearance and function. To our knowledge, this is the first study demonstrating a high correlation between clinical scar assessment and scar histology, wound contraction and re-epithelialisation data on porcine burn scars. We believe that the successful use of porcine scar scales is invaluable for assessing potential human burn treatments.

Systematic evaluation of GoSoapBox in tertiary education : a student response system for improving learning experiences and outcomes

There is currently a wide range of research into the recent introduction of student response systems in higher education and tertiary settings (Banks 2006; Kay and Le Sange, 2009; Beatty and Gerace 2009; Lantz 2010; Sprague and Dahl 2009). However, most of this pedagogical literature has generated ‘how to’ approaches regarding the use of ‘clickers’, keypads, and similar response technologies. There are currently no systematic reviews on the effectiveness of ‘GoSoapBox’ – a more recent, and increasingly popular student response system – for its capacity to enhance critical thinking, and achieve sustained learning outcomes. With rapid developments in teaching and learning technologies across all undergraduate disciplines, there is a need to obtain comprehensive, evidence-based advice on these types of technologies, their uses, and overall efficacy. This paper addresses this current gap in knowledge. Our teaching team, in an undergraduate Sociology and Public Health unit at the Queensland University of Technology (QUT), introduced GoSoapBox as a mechanism for discussing controversial topics, such as sexuality, gender, economics, religion, and politics during lectures, and to take opinion polls on social and cultural issues affecting human health. We also used this new teaching technology to allow students to interact with each other during class – both on both social and academic topics – and to generate discussions and debates during lectures. The paper reports on a data-driven study into how this interactive online tool worked to improve engagement and the quality of academic work produced by students. This paper will firstly, cover the recent literature reviewing student response systems in tertiary settings. Secondly, it will outline the theoretical framework used to generate this pedagogical research. In keeping with the social and collaborative features of Web 2.0 technologies, Bandura’s Social Learning Theory (SLT) will be applied here to investigate the effectiveness of GoSoapBox as an online tool for improving learning experiences and the quality of academic output by students. Bandura has emphasised the Internet as a tool for ‘self-controlled learning’ (Bandura 2001), as it provides the education sector with an opportunity to reconceptualise the relationship between learning and thinking (Glassman & Kang 2011). Thirdly, we describe the methods used to implement the use of GoSoapBox in our lectures and tutorials, and which aspects of the technology we drew on for learning purposes, as well as the methods for obtaining feedback from the students about the effectiveness or otherwise of this tool. Fourthly, we report cover findings from an examination of all student/staff activity on GoSoapBox as well as reports from students about the benefits and limitations of it as a learning aid. We then display a theoretical model that is produced via an iterative analytical process between SLT and our data analysis for use by academics and teachers across the undergraduate curriculum. The model has implications for all teachers considering the use of student response systems to improve the learning experiences of their students. Finally, we consider some of the negative aspects of GoSoapBox as a learning aid.

The spatialized practices of teaching writing in Australian elementary schools : diverse students shaping discoursal selves

This paper discusses the teaching of writing within the competing and often contradictory spaces of high-stakes testing and the practices and priorities around writing pedagogy in diverse school communities. It uses socio-spatial theory to examine the real-and-imagined spaces (Soja, 1996) that influence and are influenced by teachers’ pedagogical priorities for writing in two linguistically diverse elementary school case studies. Methods of critical discourse analysis are used to examine rich data sets to make visible the discourses and power relations at play in the case schools. Findings show that when teachers’ practices focus on the teaching of structure and skills alongside identity building and voice, students with diverse linguistic backgrounds can produce dramatic, authoritative and resonant texts. The paper argues that “thirdspaces” can be forged that both attend to accountability requirements, yet also give the necessary attention to more complex aspects of writing necessary for students from diverse and multilingual backgrounds to invest in writing as a creative and critical form of communication for participation in society and the knowledge economy.

An evaluation of clinical procedures used in dental implant treatment in posterior maxilla using flapless technique

This study was a measure forward in cultivating the scientific basis for an approach to examine clinical procedure in Flapless dental implant surgery. The thesis is based on: the systematic review, retrospective study of flapless implants, and in vivo study on the osseo-integration in osteoporotic rats. Dr Doan investigated "clinical procedures used in dental implant treatment in posterior maxilla using flapless technique". The work has yielded significant contributions to the area of implant flapless surgery and its effects on osteoporotic patients having implants in the posterior maxilla.

An evaluation of the Australian ‘reconceptualising early mathematics learning’ project : key findings and implications

The Pattern and Structure Mathematics Awareness Project (PASMAP) has investigated the development of patterning and early algebraic reasoning among 4 to 8 year olds over a series of related studies. We assert that an awareness of mathematical pattern and structure (AMPS) enables mathematical thinking and simple forms of generalization from an early age. This paper provides an overview of key findings of the Reconceptualizing Early Mathematics Learning empirical evaluation study involving 316 Kindergarten students from 4 schools. The study found highly significant differences on PASA scores for PASMAP students. Analysis of structural development showed increased levels for the PASMAP students; those categorised as low ability developed improved structural responses over a short period of time.

Perception Quality Evaluation with Visual and Infrared Cameras in Challenging Environmental Conditions

This work aims to contribute to the reliability and integrity of perceptual systems of unmanned ground vehicles (UGV). A method is proposed to evaluate the quality of sensor data prior to its use in a perception system by utilising a quality metric applied to heterogeneous sensor data such as visual and infrared camera images. The concept is illustrated specifically with sensor data that is evaluated prior to the use of the data in a standard SIFT feature extraction and matching technique. The method is then evaluated using various experimental data sets that were collected from a UGV in challenging environmental conditions, represented by the presence of airborne dust and smoke. In the first series of experiments, a motionless vehicle is observing a ’reference’ scene, then the method is extended to the case of a moving vehicle by compensating for its motion. This paper shows that it is possible to anticipate degradation of a perception algorithm by evaluating the input data prior to any actual execution of the algorithm.