Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

Evaluation of brief dietary questions to estimate vegetable and fruit consumption – using serum carotenoids and red-cell folate

Objective: To evaluate responses to self-administered brief questions regarding consumption of vegetables and fruit by comparison with blood levels of serum carotenoids and red-cell folate. Design: A cross-sectional study in which participants reported their usual intake of fruit and vegetables in servings per day, and serum levels of five carotenoids (α-carotene, β-carotene, β-cryptoxanthin, lutein/zeaxanthin and lycopene) and red-cell folate were measured. Serum carotenoid levels were determined by high-performance liquid chromatography, and red-cell folate by an automated immunoassay system. Settings and subjects: Between October and December 2000, a sample of 1598 adults aged 25 years and over, from six randomly selected urban centres in Queensland, Australia, were examined as part of a national study conducted to determine the prevalence of diabetes and associated cardiovascular risk factors. Results: Statistically significant (P<0.01) associations with vegetable and fruit intake (categorised into groups: ≤1 serving, 2–3 servings and ≥4 servings per day) were observed for α-carotene, β-carotene, β-cryptoxanthin, lutein/zeaxanthin and red-cell folate. The mean level of these carotenoids and of red-cell folate increased with increasing frequency of reported servings of vegetables and fruit, both before and after adjusting for potential confounding factors. A significant association with lycopene was observed only for vegetable intake before adjusting for confounders. Conclusions: These data indicate that brief questions may be a simple and valuable tool for monitoring vegetable and fruit intake in this population.

Design thinking and management education : benefits for problem framing and problem solving

Design and design thinking are identified as making valuable contributions to business and management. The numbers of higher education programs that teach design thinking to business students and executives are growing, however to date little information about the outcomes of these initiatives has emerged. This paper presents the findings from the incorporation of design thinking and methods in one unit of an MBA program. All 90 participants from three MBA classes wholeheartedly expressed their support for this initiative. An evaluation of this experiment found positive reactions, learning, changes in behaviour and positive results for their companies. The challenges and future directions for the inclusion of design thinking and design methods in management education programs are proposed.

A methodology for effective implementation of lean strategies and its performance evaluation in manufacturing organizations

Purpose – The purpose of this paper is to develop an effective methodology for implementing lean manufacturing strategies and a leanness evaluation metric using continuous performance measurement (CPM). Design/methodology/approach – Based on five lean principles, a systematic lean implementation methodology for manufacturing organizations has been proposed. A simplified leanness evaluation metric consisting of both efficiency and effectiveness attributes of manufacturing performance has been developed for continuous evaluation of lean implementation. A case study to validate the proposed methodology has been conducted and proposed CPM metric has been used to assess the manufacturing leanness. Findings – Proposed methodology is able to systematically identify manufacturing wastes, select appropriate lean tools, identify relevant performance indicators, achieve significant performance improvement and establish lean culture in the organization. Continuous performance measurement matrices in terms of efficiency and effectiveness are proved to be appropriate methods for continuous evaluation of lean performance. Research limitations/implications – Effectiveness of the method developed has been demonstrated by applying it in a real life assembly process. However, more tests/applications will be necessary to generalize the findings. Practical implications – Results show that applying the methods developed, managers can successfully identify and remove manufacturing wastes from their production processes. By improving process efficiency, they can optimize their resource allocations. Manufacturers now have a validated step by step methodology for successfully implementing lean strategies. Originality/value – According to the authors’ best knowledge, this is the first known study that proposed a systematic lean implementation methodology based on lean principles and continuous improvement techniques. Evaluation of performance improvement by lean strategies is a critical issue. This study develops a simplified leanness evaluation metric considering both efficiency and effectiveness attributes and integrates it with the lean implementation methodology.

Evaluation framework for green procurement in road construction

Purpose Most barriers and enablers of sustainable projects are related to procurement. This study proposes a framework for evaluating green procurement practices throughout the lifecycle of road construction projects and demonstrates its application through an Australian case study. Design/methodology/approach The study is based on linking the phases of road construction with incentive mechanisms for proactively motivating behavioural change. A holistic view on utilised and potential incentives is attempted with a literature review and a state-of-practice review. The latter is based on interviews and 90 policy and procurement documents across five Australian states. Findings An evaluation framework with seven procurement stages is suggested to describe current state green procurement incentives throughout the delivery lifecycle of road construction projects. The Australian case study was found to provide useful data to identify gaps and strong points of the different states regarding their level of integration of sustainability and greenhouse gas emissions GHG) reduction elements in their procurement practices. This understanding was used to draw recommendations on future advancement of green procurement. Originality/value: Government entities across the globe can impact considerably the achievement of sustainability and GHG targets, by using their procurement practices and requirements to create incentives for contractors and suppliers to engage in more GHG conscious practices. The present study provides a systematic account of how green procurement practices can be underpinned using the Australian road construction industry as a case study, and distinguish between strong and weak links in the green procurement chain to draw recommendations for future initiatives.

Longitudinal assessment of neuropathy in type 1 diabetes using novel ophthalmic markers (LANDMark) : study design and baseline characteristics

Aims Corneal nerve morphology and corneal sensation threshold have recently been explored as potential surrogate markers for the evaluation of diabetic neuropathy. We present the baseline findings of the ‘Longitudinal Assessment of Neuropathy in type 1 Diabetes using novel ophthalmic Markers’(LANDMark) study. Methods The LANDMark study is a 4-year, two-site, natural history study of three participant groups: type 1 diabetes with neuropathy (T1W), type 1 diabetes without neuropathy (T1WO) and control participants without diabetes or neuropathy. All participants undergo a detailed annual assessment of neuropathy including corneal nerve parameters measured using corneal confocal microscopy and corneal sensitivity measured using non-contact corneal aesthesiometry. Results 76 T1W, 166 T1WO and 154 control participants were enrolled into the study. Corneal sensation threshold (mbars) was significantly higher (i.e. sensitivity was lower) in T1W (1.0 ± 1.1) than T1WO (0.7 ± 0.7) and controls (0.6 ± 0.4) (p < 0.001), with no difference between T1WO and controls. Corneal nerve fibre length was lower in T1W (14.0 ± 6.4 mm/mm2) compared to T1WO (19.1 ± 5.8 mm/mm2) and controls (23.2 ± 6.3 mm/mm2) (p < 0.001). Corneal nerve fibre length was lower in T1WO compared to controls. Conclusions The LANDMark baseline findings confirm a reduction in corneal sensitivity only in Type 1 patients with neuropathy. However, corneal nerve fibre length is reduced even in Type 1 patients without neuropathy with an even greater deficit in Type 1 patients with neuropathy.

Blast response and safety evaluation of a composite column for use as key element in structural systems

This paper presents the blast response, damage mechanism and evaluation of residual load capacity of a concrete–steel composite (CSC) column using dynamic computer simulation techniques. This study is an integral part of a comprehensive research program which investigated the vulnerability of structural framing systems to catastrophic and progressive collapse under blast loading and is intended to provide design information on blast mitigation and safety evaluation of load bearing vulnerable columns that are key elements in a building. The performance of the CSC column is compared with that of a reinforced concrete (RC) column with the same dimensions and steel ratio. Results demonstrate the superior performance of the CSC column, compared to the RC column in terms of residual load carrying capacity, and its potential for use as a key element in structural systems. The procedure and results presented herein can be used in the design and safety evaluation of key elements of multi-storey buildings for mitigating the impact of blast loads.

Design Pressure Test 2013 Workshop

Design Pressure Test 2013 was a full-day intensive design immersion creative event run on Saturday 3 August 2013, at the QUT Faculty of Creative Industries J Block Design Lab Workshop in Brisbane, Australia, for 25 self-selected high-achieving junior and middle school (year 5-9) students, as part of the Queensland Academies ‘Young Scholars’ Program. Facilitated by tertiary interior design, fashion design and industrial design educators, technicians and six tertiary interior design and fashion design students, the workshop explored design process, environmental impact, the material properties and structural integrity of cardboard, construction techniques, and the production and evaluation of furniture design prototypes. This action research study aimed to facilitate an awareness in young people, of the role and scope of design within our society, the environmental ramifications of design decisions, and the value of design thinking skills in generating strategies to solve basic to complex challenges. It also aimed to investigate the value of collaboration between junior and middle school students, tertiary design educators and students and industry professionals in design awareness, and inspiring post-secondary pathways and idea generation for education. During the creative event, students utilised mathematics skills and developed sketching, making, communication, presentation and collaboration skills to improve their design process, while considering social, cultural and environmental opportunities. Through a series of hands-on collaborative design experiments, participants explored in teams of five, the opportunities available using cardboard as a material – inspiring both functional and aesthetic design solutions. Underpinned by the State Library of Queensland Design Minds Website ‘inquire, ideate and implement’ model of design thinking, the experiments culminated in the development of a detailed client brief, the design and fabrication of a furniture item for seating, and then a team presentation of prototypes to a panel of judges from the professions of architecture, interior design and industrial design, viewed also by parents. The final test for structural integrity was measured by the hoisting down of an adult body weight onto the fabricated seat. The workshop was filmed for the television program ‘Totally Wild’ for dissemination nationally (over 200,000 viewing audience) of the value of design and the Design Minds model to a wider target youth audience.

Design-build contractor selection for public sustainable buildings

Design-build (DB) is regarded as an effective means of delivering high performance green buildings, and the selection of DB contractors is of critical importance. The objective of this study is to evaluate the selection of design-builders for public buildings seeking Leadership in Energy and Environmental Design (LEED) certification and compare the selection practices involved with those of non-LEED-seeking DB projects through a robust content analysis of 74 DB request for proposals (RFPs) for public DB projects. The results of the content analysis reveal that the level of LEED certification is the dominant means of conveying the sustainability requirements in RFPs for contractor selection, with the majority of RFPs (60%) including sustainability requirements as part of the contractor evaluation package. With the exception of contractors' past performance, there is no statistically significant difference in the importance weightings of selection criteria between LEED-seeking and non-LEED-seeking buildings, and DB owners tend to place more emphasis on innovative technical solutions rather than the past performance of DB contractors. Additionally, the research findings also indicate that owners of LEED-seeking building projects tend to provide less design decisions in RFPs in order to solicit innovative design alternatives from potential DB contractors. This study provides DB owners with a number of practical implications for selecting appropriate design-builders for green DB projects.

How public owners communicate the sustainability requirements of green design-build projects

The design-build (DB) system is regarded as an effective means of delivering sustainable buildings. Specifying clear sustainability requirements to potential contractors is of great importance to project success. This research investigates the current state-of-the-practice for the definition of sustainability requirements within the public sectors of the U.S. construction market using a robust content analysis of 49 DB requests for proposals (RFPs). The results reveal that owners predominantly communicate their desired level of sustainability through the LEED certification system. The sustainability requirement has become an important dimension for the best-value evaluation of DB contractors with specific importance weightings of up to 25%. Additionally, owners of larger projects and who provide less design information in their RFPs generally allocate significantly higher importance weightings to sustainability requirements. The primary knowledge contribution of this study to the construction industry is the reveal of current trend in DB procurement for green projects. The findings also provide owners, architects, engineers, and constructors with an effective means of communicating sustainability objectives in solicitation documents.

Reframe : QUT's evaluation framework survey business rules

Reframe is changing our approach to the evaluation of courses, units, teaching and student experience at QUT. We are moving away from a single survey tool to a richer, more holistic and customisable approach. This approach will help our academics design and deliver high-quality learning experiences, and review the impact of their teaching practice on student learning. Through it, we will also be able to provide more timely access to specialised support and meet external reporting requirements.

Design of a garment for data collection of toddler language and physical activity

Design process phases of development, evaluation and implementation were used to create a garment to simultaneously collect reliable data of speech production and intensity of movement of toddlers (18-36 months). A series of prototypes were developed and evaluated that housed accelerometer-based motion sensors and a digital transmitter with microphone. The approved test garment was a top constructed from loop-faced fabric with interior pockets to house devices. Extended side panels allowed for sizing. In total, 56 toddlers (28 male; 28 female; 16-36 months of age) participated in the study providing pilot and baseline data. The test garment was effective in collecting data as evaluated for accuracy and reliability using ANOVA for accelerometer data, transcription of video for type of movement, and number and length of utterances for speech production. The data collection garment has been implemented in various studies across disciplines.

Using Participatory Design (PD) as youth engagement strategy to inform design and development of evidence-based wellbeing e-tools

The paper presents a participatory design research framework as a primary method for structuring youth engagement, participation and contribution to the design, development and usability evaluation of three evidencebased e-tools for wellbeing, which include smart phone mobile apps as well as e-health websites. The three projects are part of a series of six e-tools part of Safe and Supportive program under Young and Well CRC. The participatory design method, developed by Zelenko (2012) for application in design of online health promoting technologies, was further piloted in partnership with Inspire USA for specific application within the CRC, deploying a combination of creative design workshops and speculative design activities in developing e-tool prototypes with young people. This paper presents the resulting participatory research framework as it was implemented across the e-tool projects to facilitate active youth participation in co-designing the e-tools and ensuring the final designs are relevant to young people and deliver health messages in engaging ways. The principles of Participatory Design (PD) that inform the new framework include a high degree of participant agency in creative decisionmaking and a commitment to the process of co-designing, with young people working alongside designers and developers. The paper will showcase how the PD framework was applied across three projects to increase young people’s contribution to final design outcome.

Evaluation of a community-based intervention to promote physical activity in youth : lessons from active winners

Purpose To test the effects of a community-based physical activity intervention designed to increase physical activity and to conduct an extensive process evaluation of the intervention. Design Quasi-experimental. Setting Two rural communities in South Carolina. One community received the intervention, and the other served as the comparison. Subjects Public school students who were in fifth grade at the start of the study (558 at baseline) were eligible to participate. A total of 436 students participated over the course of the study. Intervention The intervention included after-school and summer physical activity programs and home, school, and community components designed to increase physical activity in youth. The intervention took place over an 18-month period. Measures. Students reported after-school physical activity at three data collection points (prior to, during, and following the intervention) using the Previous Day Physical Activity Recall (PDPAR). They also completed a questionnaire designed to measure hypothesized psychosocial and environmental determinants of physical activity behavior The process evaluation used meeting records, documentation of program activities, interviews, focus groups, and heart rate monitoring to evaluate the planning and implementation of the intervention. Results There were no significant differences in the physical activity variables and few significant differences in the psychosocial variables between the intervention and comparison groups. The process evaluation indicated that the after-school and summer physical activity component of the intervention was implemented as planned, but because of resource and time limitations, the home, school, and community components were not implemented as planned. Conclusions The intervention did not have a significant effect on physical activity in the target population of children in the intervention community. This outcome is similar to that reported in other studies of community-based physical activity intervention.

A comparative analysis of the conservative bridge transition design approaches

Differential settlement at the bridge approach between the deck and rail track on ground is often considered as a source of challenging technical and economical problem. This caused by the sudden stiffness changes between the bridge deck and the track on ground, and changes in soil stiffness of backfill and sub-grade with soil moisture content and loading history. To minimise the negative social and economic impacts due to poor performances of railway tracks at bridge transition zones, it is important, a special attention to be given at design, construction and maintenance stages. It is critically challenging to obtain an appropriate design solution for any given site condition and most of the existing conventional design approaches are unable to address the actual on-site behaviour due to their inherent assumptions of continuity and lack of clarifying of the local effects. An evaluation of existing design techniques is considered to estimate their contributions to a potential solution for bridge transition zones. This paper analyses five different approaches: the Chinese Standard, the European Standard with three different approaches, and the Australian approach. Each design approach is used to calculate the layer thicknesses, accounting critical design features such as the train speed, the axle load, the backfill subgrade condition, and the dynamic loading response. Considering correlation between track degradation and design parameters, this paper concludes that there is still a need of an optimised design approach for bridge transition zones.