Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

Developing sustainable ‘liveable’ buildings, cities and communities for a sub-tropical context : residents’ perspectives

A key challenge for the 21st Century is to make our cities more liveable and foster economically sustainable, environmentally responsible, and socially inclusive communities. Design thinking, particularly a human-centred approach, offers a way to tackle this challenge. Findings from two recent Australian research projects highlight how facilitating sustainable, liveable communities in a humid sub-tropical environment requires an in-depth understanding of people’s perspectives, experiences and practices. Project 1 (‘Research House’) documents the reflections of a family who lived in a ‘test’ sustainable house for two years, outlining their experience and evaluations of universal design and sustainable technologies. The study family was very impressed with the natural lighting, natural ventilation, spaciousness and ease of access, which contributed significantly to their comfort and the liveability of their home. Project 2 (‘Inner-Urban High Density Living’) explored Brisbane residents’ opinions about high-density living, through a survey (n=636), interviews (n=24), site observations (over 300 hours) and environmental monitoring, assessing opinions on the liveability of their individual dwelling, the multi-unit host building and the surrounding neighbourhood. Nine areas, categorised into three general domains, were identified as essential for enhancing high density liveability. In terms of the dwelling, thermal comfort/ventilation, natural light, noise mitigation were important; shared space, good neighbour protocols, and support for environmentally sustainable behaviour were desired in the building/complex; and accessible/sustainable transport, amenities and services, sense of community were considered important in the surrounding neighbourhood. Combined, these findings emphasise the importance and complexity associated with designing liveable building, cities and communities, illustrating how adopting a design thinking, human-centred approach will help create sustainable communities that will meet the needs of current and future generations.

Technological tools for science classrooms : choosing and using for productive and sustainable teaching and learning experiences

In this age of rapidly evolving technology, teachers are encouraged to adopt ICTs by government, syllabus, school management, and parents. Indeed, it is an expectation that teachers will incorporate technologies into their classroom teaching practices to enhance the learning experiences and outcomes of their students. In particular, regarding the science classroom, a subject that traditionally incorporates hands-on experiments and practicals, the integration of modern technologies should be a major feature. Although myriad studies report on technologies that enhance students’ learning outcomes in science, there is a dearth of literature on how teachers go about selecting technologies for use in the science classroom. Teachers can feel ill prepared to assess the range of available choices and might feel pressured and somewhat overwhelmed by the avalanche of new developments thrust before them in marketing literature and teaching journals. The consequences of making bad decisions are costly in terms of money, time and teacher confidence. Additionally, no research to date has identified what technologies science teachers use on a regular basis, and whether some purchased technologies have proven to be too problematic, preventing their sustained use and possible wider adoption. The primary aim of this study was to provide research-based guidance to teachers to aid their decision-making in choosing technologies for the science classroom. The study unfolded in several phases. The first phase of the project involved survey and interview data from teachers in relation to the technologies they currently use in their science classrooms and the frequency of their use. These data were coded and analysed using Grounded Theory of Corbin and Strauss, and resulted in the development of a PETTaL model that captured the salient factors of the data. This model incorporated usability theory from the Human Computer Interaction literature, and education theory and models such as Mishra and Koehler’s (2006) TPACK model, where the grounded data indicated these issues. The PETTaL model identifies Power (school management, syllabus etc.), Environment (classroom / learning setting), Teacher (personal characteristics, experience, epistemology), Technology (usability, versatility etc.,) and Learners (academic ability, diversity, behaviour etc.,) as fields that can impact the use of technology in science classrooms. The PETTaL model was used to create a Predictive Evaluation Tool (PET): a tool designed to assist teachers in choosing technologies, particularly for science teaching and learning. The evolution of the PET was cyclical (employing agile development methodology), involving repeated testing with in-service and pre-service teachers at each iteration, and incorporating their comments i ii in subsequent versions. Once no new suggestions were forthcoming, the PET was tested with eight in-service teachers, and the results showed that the PET outcomes obtained by (experienced) teachers concurred with their instinctive evaluations. They felt the PET would be a valuable tool when considering new technology, and it would be particularly useful as a means of communicating perceived value between colleagues and between budget holders and requestors during the acquisition process. It is hoped that the PET could make the tacit knowledge acquired by experienced teachers about technology use in classrooms explicit to novice teachers. Additionally, the PET could be used as a research tool to discover a teachers’ professional development needs. Therefore, the outcomes of this study can aid a teacher in the process of selecting educationally productive and sustainable new technology for their science classrooms. This study has produced an instrument for assisting teachers in the decision-making process associated with the use of new technologies for the science classroom. The instrument is generic in that it can be applied to all subject areas. Further, this study has produced a powerful model that extends the TPACK model, which is currently extensively employed to assess teachers’ use of technology in the classroom. The PETTaL model grounded in data from this study, responds to the calls in the literature for TPACK’s further development. As a theoretical model, PETTaL has the potential to serve as a framework for the development of a teacher’s reflective practice (either self evaluation or critical evaluation of observed teaching practices). Additionally, PETTaL has the potential for aiding the formulation of a teacher’s personal professional development plan. It will be the basis for further studies in this field.

Evaluation of two-view geometry methods with automatic ground-truth generation

A large number of methods have been published that aim to evaluate various components of multi-view geometry systems. Most of these have focused on the feature extraction, description and matching stages (the visual front end), since geometry computation can be evaluated through simulation. Many data sets are constrained to small scale scenes or planar scenes that are not challenging to new algorithms, or require special equipment. This paper presents a method for automatically generating geometry ground truth and challenging test cases from high spatio-temporal resolution video. The objective of the system is to enable data collection at any physical scale, in any location and in various parts of the electromagnetic spectrum. The data generation process consists of collecting high resolution video, computing accurate sparse 3D reconstruction, video frame culling and down sampling, and test case selection. The evaluation process consists of applying a test 2-view geometry method to every test case and comparing the results to the ground truth. This system facilitates the evaluation of the whole geometry computation process or any part thereof against data compatible with a realistic application. A collection of example data sets and evaluations is included to demonstrate the range of applications of the proposed system.

An early assessment of silicone hydrogel safety : pearls and pitfalls, and current status

The contact lens industry has evolved and now provides many choices, including continuous wear, overnight orthokeratology, frequent-replacement lenses, daily-disposable lenses, and many alternatives in systems of care and maintenance. Epidemiologic studies to date have shown that how a lens is worn, particularly if worn overnight, can increase the risk of microbial keratitis. However, the risk of silicone hydrogel contact lenses worn on a continuous-wear basis has been evaluated only recently. This article summarizes the recent research data on extended-wear silicone hydrogel lenses and discusses the challenges of early evaluations of silicone hydrogel lens safety. Finally, the relevance of this information is discussed to practitioners and contact lens wearers making choices about the risks and benefits of different products and how they are used.

The impact on clinical practice of a postgraduate clinical pharmacy programme, incorporating competency-based performance evaluation

Background A goal of the postgraduate clinical pharmacy programme (PGCPP) at the University of Queensland is to enhance clinical practice. Aims To evaluate student perceptions of the impact of the PGCPP on practice and the inclusion of a competency-based performance evaluation as a formative component of the curriculum. Method In 2010, students completed a questionnaire to evaluate the impact of the PGCPP. In 2011, formative competency-based performance evaluations were conducted as a component of the course and the questionnaire was repeated. Responses, competency ratings and evaluation feedback were collated. Data were analysed using descriptive statistics. Results 51/57 (89%) of students completed the questionnaire in 2010 and 2011. Over 90% of students agreed or strongly agreed that the PGCPP enhanced practice, knowledge, confidence and contribution to patient care. Responses were similarly positive after the inclusion of the performance evaluation. Conclusion This study demonstrated that the PGCPP is achieving the goal of enhancing the practice of pharmacists.

Oscar Pistorius and the death of Reeva Steenkamp : mapping public opinions on Twitter using frame analysis and data mining

This paper uses innovative content analysis techniques to map how the death of Oscar Pistorius' girlfriend, Reeva Steenkamp, was framed on Twitter conversations. Around 1.5 million posts from a two-week timeframe are analyzed with a combination of syntactic and semantic methods. This analysis is grounded in the frame analysis perspective and is different than sentiment analysis. Instead of looking for explicit evaluations, such as “he is guilty” or “he is innocent”, we showcase through the results how opinions can be identified by complex articulations of more implicit symbolic devices such as examples and metaphors repeatedly mentioned. Different frames are adopted by users as more information about the case is revealed: from a more episodic one, highly used in the very beginning, to more systemic approaches, highlighting the association of the event with urban violence, gun control issues, and violence against women. A detailed timeline of the discussions is provided.

Nanohybrids of silver particles on clay platelets delaminate Pseudomonas biofilms

Aim To evaluate the effectiveness of novel nanohybrids, composed of silver nanoparticles and nanoscale silicate platelets, to clear Pseudomonas aeruginosa biofilms. Materials & methods The nanohybrids were manufactured from an in situ reduction of silver salts in the silicate platelet dispersion, and then applied to biofilms in vitro and in vivo. Results In reference to the biocidal effects of gentamycin, the nanohybrids mitigated the spreading of the biofilms, and initiated robust cell death and exfoliation from the superficial layers of the biofilms in vitro. In vivo, the nanohybrids exhibited significant therapeutic effects by eliminating established biofilms from infected corneas and promoting the recovery of corneal integrity. Conclusion All of the evaluations indicate the high potency of the newly developed silver nanoparticle/nanoscale silicate platelet nanohybrids for eliminating biofilms.

Mechanism based emulation of dynamic simulation models : concept and application in hydrology

Many model-based investigation techniques, such as sensitivity analysis, optimization, and statistical inference, require a large number of model evaluations to be performed at different input and/or parameter values. This limits the application of these techniques to models that can be implemented in computationally efficient computer codes. Emulators, by providing efficient interpolation between outputs of deterministic simulation models, can considerably extend the field of applicability of such computationally demanding techniques. So far, the dominant techniques for developing emulators have been priors in the form of Gaussian stochastic processes (GASP) that were conditioned with a design data set of inputs and corresponding model outputs. In the context of dynamic models, this approach has two essential disadvantages: (i) these emulators do not consider our knowledge of the structure of the model, and (ii) they run into numerical difficulties if there are a large number of closely spaced input points as is often the case in the time dimension of dynamic models. To address both of these problems, a new concept of developing emulators for dynamic models is proposed. This concept is based on a prior that combines a simplified linear state space model of the temporal evolution of the dynamic model with Gaussian stochastic processes for the innovation terms as functions of model parameters and/or inputs. These innovation terms are intended to correct the error of the linear model at each output step. Conditioning this prior to the design data set is done by Kalman smoothing. This leads to an efficient emulator that, due to the consideration of our knowledge about dominant mechanisms built into the simulation model, can be expected to outperform purely statistical emulators at least in cases in which the design data set is small. The feasibility and potential difficulties of the proposed approach are demonstrated by the application to a simple hydrological model.

Evaluating leadership development using the Most Significant Change technique

Course evaluations are now a serious matter for universities trying to meet stakeholder needs and expectations, quality assurance, improvements and strategic decision making. Typically, students are invited to participate in surveys on how well the design and delivery aspects meet predetermined learning objectives, quality of teaching, and the types of improvements needed for future deliveries. We used the Most Significant Change technique to gather data on the impact of a leadership course on 18 Pacific Islanders who completed a Master of Education (Educational Leadership). Participants' views highlighted impacts that were of significance to the students and their workplaces. The findings demonstrate that the Most Significant Change technique offers a more comprehensive understanding of the impact of leadership development courses.

Talk2Me : engaging interactive installation environments

In this research I explore what elements there may be in common between tangible interactive-technology works that successfully engage their participants. An exploration of existing methods for obtaining useful evaluations for non-use and ambiguous environments forms a part of the discussion.

Technique for evaluating photo sharing interfaces with the early prototypes - group simulation

User evaluations using paper prototypes commonly lack social context. The Group simulation technique described in this paper offers a solution to this problem. The study introduces an early-phase participatory design technique targeted for small groups. The proposed technique is used for evaluating an interface, which enables group work in photo collection creation. Three groups of four users, 12 in total, took part in a simulation session where they tested a low-fidelity design concept that included their own personal photo content from an event that their group attended together. The users’ own content was used to evoke natural experiences. Our results indicate that the technique helped users to naturally engage with the prototype in the session. The technique is suggested to be suitable for evaluating other early-phase concepts and to guide design solutions, especially with the concepts that include users’ personal content and enable content sharing.

Quantitative estimation of bioturbation based on digital image analysis

Quantitative determination of modification of primary sediment features, by the activity of organisms (i.e., bioturbation) is essential in geosciences. Some methods proposed since the 1960s are mainly based on visual or subjective determinations. The first semiquantitative evaluations of the Bioturbation Index, Ichnofabric Index, or the amount of bioturbation were attempted, in the best cases using a series of flashcards designed in different situations. Recently, more effective methods involve the use of analytical and computational methods such as X-rays, magnetic resonance imaging or computed tomography; these methods are complex and often expensive. This paper presents a compilation of different methods, using Adobe® Photoshop® software CS6, for digital estimation that are a part of the IDIAP (Ichnological Digital Analysis Images Package), which is an inexpensive alternative to recently proposed methods, easy to use, and especially recommended for core samples. The different methods — “Similar Pixel Selection Method (SPSM)”, “Magic Wand Method (MWM)” and the “Color Range Selection Method (CRSM)” — entail advantages and disadvantages depending on the sediment (e.g., composition, color, texture, porosity, etc.) and ichnological features (size of traces, infilling material, burrow wall, etc.). The IDIAP provides an estimation of the amount of trace fossils produced by a particular ichnotaxon, by a whole ichnocoenosis or even for a complete ichnofabric. We recommend the application of the complete IDIAP to a given case study, followed by selection of the most appropriate method. The IDIAP was applied to core material recovered from the IODP Expedition 339, enabling us, for the first time, to arrive at a quantitative estimation of the discrete trace fossil assemblage in core samples.

Crowdsourcing interactions : using crowdsourcing for evaluating interactive information retrieval systems

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.

Considering the economic value of natural design elements at city scale

With increasing signs of climate change and the influence of national and international carbon-related laws and agreements, governments all over the world are grappling with how to rapidly transition to low-carbon living. This includes adapting to the impacts of climate change that are very likely to be experienced due to current emission levels (including extreme weather and sea level changes), and mitigating against further growth in greenhouse gas emissions that are likely to result in further impacts. Internationally, the concept of ‘Biophilic Urbanism’, a term coined by Professors Tim Beatley and Peter Newman to refer to the use of natural elements as design features in urban landscapes, is emerging as a key component in addressing such climate change challenges in rapidly growing urban contexts. However, the economics of incorporating such options is not well understood and requires further attention to underpin a mainstreaming of biophilic urbanism. Indeed, there appears to be an ad hoc, reactionary approach to creating economic arguments for or against the design, installation or maintenance of natural elements such as green walls, green roofs, streetscapes, and parklands. With this issue in mind, this paper will overview research as part of an industry collaborative research project that considers the potential for using a number of environmental economic valuation techniques that have evolved over the last several decades in agricultural and resource economics, to systematically value the economic value of biophilic elements in the urban context. Considering existing literature on environmental economic valuation techniques, the paper highlights opportunities for creating a standardised language for valuing biophilic elements. The conclusions have implications for expanding the field of environmental economic value to support the economic evaluations and planning of the greater use of natural elements in cities. Insights are also noted for the more mature fields of agricultural and resource economics.