Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

Sustainability of a successful health and nutrition program in a remote Aboriginal community

OBJECTIVE: To assess the long term effect of a nutrition program in a remote Aboriginal community (Minjilang). DESIGN: Evaluation of nutritional outcomes over the three years before and the three years after a health and nutrition program that ran from June 1989 to June 1990. Turnover of food items at the community store was used as a measure of dietary intake at Minjilang and a comparison community. SETTING: A community of about 150 Aboriginal people live at Minjilang on Croker Island, 240 km north-east of Darwin. A similar community of about 300 people on another island was used as the comparison. RESULTS: The program produced lasting improvements in dietary intake of most target foods (including fruit, vegetables and wholegrain bread) and nutrients (including folate, ascorbic acid and thiamine). Sugar intake fell in both communities before the program, but the additional decrease in sugar consumption during the program at Minjilang "rebounded" in the next year. Dietary improvements in the comparison community were delayed and smaller than at Minjilang. CONCLUSIONS: The success of the program at Minjilang was linked to an ongoing process of social change, which in turn provided a stimulus for dietary improvement in the comparison community. When Aboriginal people themselves control and maintain ownership of community-based intervention programs, nutritional improvements can be initiated and sustained.

Evaluation framework for green procurement in road construction

Purpose Most barriers and enablers of sustainable projects are related to procurement. This study proposes a framework for evaluating green procurement practices throughout the lifecycle of road construction projects and demonstrates its application through an Australian case study. Design/methodology/approach The study is based on linking the phases of road construction with incentive mechanisms for proactively motivating behavioural change. A holistic view on utilised and potential incentives is attempted with a literature review and a state-of-practice review. The latter is based on interviews and 90 policy and procurement documents across five Australian states. Findings An evaluation framework with seven procurement stages is suggested to describe current state green procurement incentives throughout the delivery lifecycle of road construction projects. The Australian case study was found to provide useful data to identify gaps and strong points of the different states regarding their level of integration of sustainability and greenhouse gas emissions GHG) reduction elements in their procurement practices. This understanding was used to draw recommendations on future advancement of green procurement. Originality/value: Government entities across the globe can impact considerably the achievement of sustainability and GHG targets, by using their procurement practices and requirements to create incentives for contractors and suppliers to engage in more GHG conscious practices. The present study provides a systematic account of how green procurement practices can be underpinned using the Australian road construction industry as a case study, and distinguish between strong and weak links in the green procurement chain to draw recommendations for future initiatives.

An evaluation of the culture performance of two Macrobrachium species in Vanuatu: the exotic M. rosenbergii and the indigenous M. lar

Fisheries and aquaculture are important for food security, income generation and are critical to long term sustainability of many countries. Freshwater prawns have been harvested in the streams and creeks in Vanuatu, however due to over-exploitation catches have declined in recent years. To satisfy high demand for this product, Vanuatu government intends to establish economically viable small-scale aquaculture industries. The current project showed that wild Macrobrachium lar in Vanuatu constitute a single population for management purposes and that M. rosenbergii grows much faster than M. lar in simple pond grow-out systems, hence is a better species for culture in Vanuatu.

Managing cost implications for highway infrastructure sustainability

Highway construction works have significant bearings on all aspects of sustainability. With the increasing level of public awareness and government regulatory measures, the construction industry is experiencing a cultural shift to recognise, embrace and pursue sustainability. Stakeholders are now keen to identify sustainable alternatives and the financial implications of including them on a lifecycle basis. They need tools that can aid the evaluation of investment options. To date, however, there have not been many financial assessments on the sustainability aspects of highway projects. This is because the existing life-cycle costing analysis (LCCA) models tend to focus on economic issues alone and are not able to deal with sustainability factors. This paper provides insights into the current practice of life-cycle cost analysis, and the identification and quantification of sustainability-related cost components in highway projects through literature review, questionnaire surveys and semi-structured interviews. The results can serve as a platform for highway project stakeholders to develop practical tools to evaluate highway investment decisions and reach an optimum balance between financial viability and sustainability deliverables.

How public owners communicate the sustainability requirements of green design-build projects

The design-build (DB) system is regarded as an effective means of delivering sustainable buildings. Specifying clear sustainability requirements to potential contractors is of great importance to project success. This research investigates the current state-of-the-practice for the definition of sustainability requirements within the public sectors of the U.S. construction market using a robust content analysis of 49 DB requests for proposals (RFPs). The results reveal that owners predominantly communicate their desired level of sustainability through the LEED certification system. The sustainability requirement has become an important dimension for the best-value evaluation of DB contractors with specific importance weightings of up to 25%. Additionally, owners of larger projects and who provide less design information in their RFPs generally allocate significantly higher importance weightings to sustainability requirements. The primary knowledge contribution of this study to the construction industry is the reveal of current trend in DB procurement for green projects. The findings also provide owners, architects, engineers, and constructors with an effective means of communicating sustainability objectives in solicitation documents.

Sustainability outcomes of infrastructure sustainability rating schemes for road projects

The construction and operation of infrastructure assets can have significant impact on society and the region. Using a sustainability assessment framework can be an effective means to build sustainability aspects into the design, construction and operation of infrastructure assets. The conventional evaluation processes and procedures for infrastructure projects do not necessarily measure the qualitative/quantitative effectiveness of all aspects of sustainability: environment, social wellbeing and economy. As a result, a few infrastructure sustainability rating schemes have been developed with a view to assess the level of sustainability attained in the infrastructure projects. These include: Infrastructure Sustainability (Australia); CEEQUAL (UK); and Envision (USA). In addition, road sector specific sustainability rating schemes such as Greenroads (USA) and Invest (Australia) have also been developed. These schemes address several aspects of sustainability with varying emphasis (weightings) on areas such as: use of resources; emission, pollution and waste; ecology; people and place; management and governance; and innovation. The attainment of sustainability of an infrastructure project depends largely on addressing the whole-of-life environmental issues. This study has analysed the rating schemes’ coverage of different environmental components for the road infrastructure under the five phases of a project: material, construction, use, maintenance and end-of-life. This is based on a comprehensive life cycle assessment (LCA) system boundary. The findings indicate that there is a need for the schemes to consider key (high impact) life cycle environmental components such as traffic congestion during construction, rolling resistance due to surface roughness and structural stiffness of the pavement, albedo, lighting, and end-of-life management (recycling) to deliver sustainable road projects.

Implementing a holistic process for embedding sustainability: A case study in first year engineering, Monash University, Australia

While there is clear recognition of the need to incorporate sustainable development into university curricula, there is limited research that examines how to achieve that integration or evaluates its impacts on student learning. This paper responds to these knowledge gaps through a case study of curriculum renewal that involved embedding sustainability into a first year engineering curriculum. The initiative was guided by a deliberative and dynamic model for curriculum renewal that brought together internal and external stakeholders through a structured sequence of facilitated workshops and meetings. That process identified sustainability-related knowledge and skills relevant for first year engineering, and faculty members teaching in the first year program were guided through a process of curriculum renewal to meet those needs. The process through which the whole of curriculum renewal was undertaken is innovative and provides a case study of precedent in the field of education for sustainability. The study demonstrates the contribution that can be made by a web-based sustainability portal in supporting curriculum renewal. Learning and teaching outcomes were evaluated through ‘before and after surveys’ of the first year engineering students. Statistically significant increases in student's self-reported knowledge of sustainability were measured as a result of exposure to the renewed first year curriculum and this confirmed the value of the initiative in terms of enhancing student learning. While applied in this case to engineering, the process to achieve integration of sustainability into the curriculum approach is likely to have value for other academic disciplines. Considering student performance on assignments and exam questions relating to sustainability would provide a stronger basis for future research to understand the impact of initiatives like this on student learning.

Finding the leverage point in the sustainability crisis: Global, national and regional Australian responses

From an economic perspective, the sustainability crisis is ultimately characterized by a worsening relationship between the resources required to support the global population and the ability of the earth to supply them. Despite the ever-increasing threat of a calamity, modern society appears unable to alter its course. The very systems which underpin global human endeavor seem to actively prevent meaningful change and the one irrepressible goal to which all societies seem to strive is the very thing that makes such endeavor ultimately life threatening: that of global growth. Using the Australian experience as an exemplar, this paper explores how the concept of growth infiltrates societal reactions to the crisis at various scales – global, national and regional. Analysis includes historic studies, a critique of current misconceptions around population demographics, comparative evaluation of various interventions in the Australian context and considerations around potential ways to address the crisis.

Managing the impacts of sugarcane farming on the Great Barrier Reef: An evaluation of the implementation of the Polluter Pays Principle

This report provides an evaluation of the implementation of the Polluter Pays Principle (PPP) – a principle of international environmental law – in the context of pollution from sugarcane farming affecting Australia’s Great Barrier Reef (GBR). The research was part of an experiment to test methods for evaluating the effectiveness of environmental laws. Overall, we found that whilst the PPP is reflected to a limited extent in Australian law (more so in Queensland law, than at the national level), the behaviour one might expect in terms of implementing the principle was largely inadequate. Evidence of a longer term, explicit commitment to the PPP was particularly weak.

Exploring profit - Sustainability trade-offs in cropping systems using evolutionary algorithms

Models that implement the bio-physical components of agro-ecosystems are ideally suited for exploring sustainability issues in cropping systems. Sustainability may be represented as a number of objectives to be maximised or minimised. However, the full decision space of these objectives is usually very large and simplifications are necessary to safeguard computational feasibility. Different optimisation approaches have been proposed in the literature, usually based on mathematical programming techniques. Here, we present a search approach based on a multiobjective evaluation technique within an evolutionary algorithm (EA), linked to the APSIM cropping systems model. A simple case study addressing crop choice and sowing rules in North-East Australian cropping systems is used to illustrate the methodology. Sustainability of these systems is evaluated in terms of economic performance and resource use. Due to the limited size of this sample problem, the quality of the EA optimisation can be assessed by comparison to the full problem domain. Results demonstrate that the EA procedure, parameterised with generic parameters from the literature, converges to a useable solution set within a reasonable amount of time. Frontier ‘‘peels’’ or Pareto-optimal solutions as described by the multiobjective evaluation procedure provide useful information for discussion on trade-offs between conflicting objectives.

Analysis of governance for sustainability planning in the Cairns Region

The relationship between governance arrangements and sustainability planning outcomes in complex governance systems remains poorly understood, despite significant discussions of governance in the environmental management literature emerging in the last decade. In order to analyse and examine the relationship between the health of sustainability planning governance and decision-making outcomes, this paper applies the Governance Systems Analysis framework (GSA) in the Cairns region. This paper analyses the sustainability planning governance arrangements in the Cairns region by exploring the capacity, connectivity and knowledge use of institutions in the region to deliver desired sustainability planning outcomes. The paper finds that the planning for sustainability in the Cairns region is on a knife’s edge, and could fail or succeed to deliver its intended decision-making outcomes. The paper concludes with recommendations for governance reform for sustainability in the Cairns region.

Reflecting on Reggio: An evaluation of design intent in an early childhood learning environment

Reggio Emilia is an educational philosophy that encourages teachers, students and their parents to collaborate and actively engage with the environment. This study investigates how the Reggio Emilia design approach was translated architecturally for a kindergarten in an Australian context, and provides insights into the operation of this Reggio kindergarten and the impact that it is now having on the occupants. It evaluates the original architectural design intent of the Reggio Emilia early childhood learning environment against its spatial provision. The relationship that the Reggio Emilia approach facilitates between students and the environment, and the contribution that this approach has on their learning, are also explored. Several key themes emerging from the Reggio values were identified in the literature. These were then used to inform an exploration of the kindergarten spaces and places.. Architects, teachers and a sustainability manager of the kindergarten were interviewed with their experiences constituting the primary data of this study. Using a Grounded Theory methodology, systematic data coding and analysis were then conducted. Themes and concepts that emerged from this process include: differing interpretations of the Reggio Emilia philosophy; motivations for neglect of traditional external structures and play equipment; the impact of education for sustainability; and the positive effects that Reggio Emilia is having on the rest of the institution’s development.

Assessing the sustainability of wheat-based cropping systems using simulation modelling: sustainability

Concepts of agricultural sustainability and possible roles of simulation modelling for characterising sustainability were explored by conducting, and reflecting on, a sustainability assessment of rain-fed wheat-based systems in the Middle East and North Africa region. We designed a goal-oriented, model-based framework using the cropping systems model Agricultural Production Systems sIMulator (APSIM). For the assessment, valid (rather than true or false) sustainability goals and indicators were identified for the target system. System-specific vagueness was depicted in sustainability polygons-a system property derived from highly quantitative data-and denoted using descriptive quantifiers. Diagnostic evaluations of alternative tillage practices demonstrated the utility of the framework to quantify key bio-physical and chemical constraints to sustainability. Here, we argue that sustainability is a vague, emergent system property of often wicked complexity that arises out of more fundamental elements and processes. A 'wicked concept of sustainability' acknowledges the breadth of the human experience of sustainability, which cannot be internalised in a model. To achieve socially desirable sustainability goals, our model-based approach can inform reflective evaluation processes that connect with the needs and values of agricultural decision-makers. Hence, it can help to frame meaningful discussions, from which actions might emerge.

Assessing the integrated sustainability of a public rental housing project from the perspective of complex eco-system

Public rental housing (PRH) projects are the mainstream of China's new affordable housing policies, and their integrated sustainability has a far-reaching effect on medium-low income families' well-being and social stability. However, there are few quantitative researches on the integrated sustainability of PRH projects. Our study tries to fill this gap through proposing an assessment model of the integrated sustainability for PRH projects. First, this paper defines what the sustainability of a PRH project is. Second, after constructing the sustainable system of a PRH project from the perspective of complex eco-system, the paper explores the internal operation mechanism and the coupling mechanism among the ecological, economic and social subsystems. Third, it identifies fourteen indices to represent the sustainability system of a PRH project, including six indices of ecological subsystem, five of economic subsystem and three of social subsystem. Fourth, it qualifies the weights of three subsystems and their internal representative indices. In addition, an assessment model is established through expert surveys and analytic network process (ANP). Finally, the paper carries out an empirical research on a PRH project in Nanjing city of China, followed by suggestions to enhance the integrated sustainability. The sustainability system and its evaluation model proposed in this paper are concise and easy to understand and can provide a theoretical foundation and a scientific basis for the evaluation and optimization of PRH projects.