A review of project evaluation methodologies for toll road projects

Project evaluation is a process of measuring costs, benefits, risks and uncertainties for the purpose of decision-making by estimating and assessing impacts of the project to the community. The effects of impacts of toll roads are similar but different from the general non-tolled roads. Project evaluation methodologies are extensively studied and applied to various transport infrastructure projects. However, there is no definitive methodology to evaluate toll roads. This review discusses the impacts of toll roads then reviews the limitations of existing project evaluation methodologies when evaluating toll road impacts. The review identified gaps of knowledge of toll evaluations. First, the treatment of toll in project evaluation, particularly in Cost-Benefit Analysis requires further study to explore the appropriate methodology. Secondly, the project evaluation methodology needs to place strong emphasis on empirically based risk and uncertainty assessment. Addressing the limitations of the existing project evaluation methodologies leads to improvements of the methodology in practical level as well as fills the gap of knowledge of project evaluation for toll roads with respect to net impacts to the community.

Overview of the INEX 2010 focused relevance feedback track

The INEX 2010 Focused Relevance Feedback track offered a refined approach to the evaluation of Focused Relevance Feedback algorithms through simulated exhaustive user feedback. As in traditional approaches we simulated a user-in-the loop by re-using the assessments of ad-hoc retrieval obtained from real users who assess focused ad-hoc retrieval submissions. The evaluation was extended in several ways: the use of exhaustive relevance feedback over entire runs; the evaluation of focused retrieval where both the retrieval results and the feedback are focused; the evaluation was performed over a closed set of documents and complete focused assessments; the evaluation was performed over executable implementations of relevance feedback algorithms; and �finally, the entire evaluation platform is reusable. We present the evaluation methodology, its implementation, and experimental results obtained for nine submissions from three participating organisations.

Overview of the INEX 2011 relevance feedback track

The INEX 2011 Relevance Feedback track offered a refined approach to the evaluation of Focused Relevance Feedback algorithms through simulated exhaustive user feedback. Run in largely identical fashion to the Relevance Feedback track in INEX 2010[2], we simulated a user-in-the loop by re-using the assessments of ad-hoc retrieval obtained from real users who assess focused ad-hoc retrieval submissions. We present the evaluation methodology, its implementation, and experimental results obtained for four submissions from two participating organisations. As the task and evaluation methods did not change between INEX 2010 and now, explanations of these details from the INEX 2010 version of the track have been repeated verbatim where appropriate.

“I understand. I am a participant”: Navigating the ‘fuzzy’ boundaries of visual methods in qualitative longitudinal research

In this chapter we discuss how utilising the participatory visual methodology, photovoice, in an aged care context with its unique communal setting raised several ‘fuzzy boundary’ ethical dilemmas. To illustrate these challenges, we draw on immersive field notes from an ongoing qualitative longitudinal research (QLR) exploring the lived experience of aged care from the perspective of older residents, and focus on interactions with one participant, 81 year old Cassie. We explore how the camera, which is integral to the photovoice method, altered the researcher/participant ethical dynamics by becoming a continual ‘connector’ to the researcher. The camera took on a distinct agency, acting as a non-threatening ‘portal’ that lengthened contact, provided informal opportunities to alter the relationship dynamics and enabled unplanned participant revelation.

Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations

There has been a greater focus on strengthening evaluation capacity building (ECB) within development organisations in recent years. This can be attributed in part to the growing appreciation of the value of participatory and collaborative forms of evaluation. Evaluation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance (Horton et al., 2003:7). While there are many benefits of using participatory methodologies in ECB projects, our experiences and a review of the literature in this area highlight the many challenges, issues and contradictions that can affect the success of such ECB efforts. We discuss these issues, drawing on our learnings from the ongoing participatory action research (PAR) project 'Assessing Communication for Social Change’ (AC4SC). This four year project, which began in 2007, is a collaboration between communication and development academics and evaluation specialists from two Australian universities and communication for development practitioners and monitoring and evaluation (M&E) staff in the NGO Equal Access Nepal (EAN). The aim is to develop, implement, and evaluate a participatory methodology for assessing the social change impacts of community radio programs produced by EAN. It builds on previous projects that used ethnographic action research (EAR) methodology (Tacchi et al., 2007).

Developing a participatory impact assessment approach and action research culture within a communication for social change organisation in Nepal

Developing an effective impact evaluation framework, managing and conducting rigorous impact evaluations, and developing a strong research and evaluation culture within development communication organisations presents many challenges. This is especially so when both the community and organisational context is continually changing and the outcomes of programs are complex and difficult to clearly identify.----- This paper presents a case study from a research project being conducted from 2007-2010 that aims to address these challenges and issues, entitled Assessing Communication for Social Change: A New Agenda in Impact Assessment. Building on previous development communication projects which used ethnographic action research, this project is developing, trailing and rigorously evaluating a participatory impact assessment methodology for assessing the social change impacts of community radio programs in Nepal. This project is a collaboration between Equal Access – Nepal (EAN), Equal Access – International, local stakeholders and listeners, a network of trained community researchers, and a research team from two Australian universities. A key element of the project is the establishment of an organisational culture within EAN that values and supports the impact assessment process being developed, which is based on continuous action learning and improvement. The paper describes the situation related to monitoring and evaluation (M&E) and impact assessment before the project began, in which EAN was often reliant on time-bound studies and ‘success stories’ derived from listener letters and feedback. We then outline the various strategies used in an effort to develop stronger and more effective impact assessment and M&E systems, and the gradual changes that have occurred to date. These changes include a greater understanding of the value of adopting a participatory, holistic, evidence-based approach to impact assessment. We also critically review the many challenges experienced in this process, including:----- • Tension between the pressure from donors to ‘prove’ impacts and the adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations.----- • Resistance from the content teams to changing their existing M&E practices and to the perceived complexity of the approach.----- • Lack of meaningful connection between the M&E and content teams.----- • Human resource problems and lack of capacity in analysing qualitative data and reporting results.----- • The contextual challenges, including extreme poverty, wide cultural and linguistic diversity, poor transport and communications infrastructure, and political instability.----- • A general lack of acceptance of the importance of evaluation within Nepal due to accepting everything as fate or ‘natural’ rather than requiring investigation into a problem.

Empirical evaluation of a service analysis and design methodology

Service-orientation has gained widespread acceptance and is increasingly being employed as a paradigm for structuring both business and IT architectures. An earlier study of extant service analysis and design methodologies discovered a need for holistic approaches that equally account for both business and software services, which motivated the design of a new, consolidated service analysis and design methodology. A challenge in design-oriented research is to evaluate the utility of the newly created artefacts (here: the methodology), as they are often intended to become part of complex socio-technical systems. Therefore, after presenting a brief overview of the consolidated methodology, the paper discusses possible approaches for the “evaluate” phase of this design-science research process and presents the results of an empirical evaluation conducted in an Action Research study at one of Australia’s largest financial services providers.

A methodology for effective implementation of lean strategies and its performance evaluation in manufacturing organizations

Purpose – The purpose of this paper is to develop an effective methodology for implementing lean manufacturing strategies and a leanness evaluation metric using continuous performance measurement (CPM). Design/methodology/approach – Based on five lean principles, a systematic lean implementation methodology for manufacturing organizations has been proposed. A simplified leanness evaluation metric consisting of both efficiency and effectiveness attributes of manufacturing performance has been developed for continuous evaluation of lean implementation. A case study to validate the proposed methodology has been conducted and proposed CPM metric has been used to assess the manufacturing leanness. Findings – Proposed methodology is able to systematically identify manufacturing wastes, select appropriate lean tools, identify relevant performance indicators, achieve significant performance improvement and establish lean culture in the organization. Continuous performance measurement matrices in terms of efficiency and effectiveness are proved to be appropriate methods for continuous evaluation of lean performance. Research limitations/implications – Effectiveness of the method developed has been demonstrated by applying it in a real life assembly process. However, more tests/applications will be necessary to generalize the findings. Practical implications – Results show that applying the methods developed, managers can successfully identify and remove manufacturing wastes from their production processes. By improving process efficiency, they can optimize their resource allocations. Manufacturers now have a validated step by step methodology for successfully implementing lean strategies. Originality/value – According to the authors’ best knowledge, this is the first known study that proposed a systematic lean implementation methodology based on lean principles and continuous improvement techniques. Evaluation of performance improvement by lean strategies is a critical issue. This study develops a simplified leanness evaluation metric considering both efficiency and effectiveness attributes and integrates it with the lean implementation methodology.

Making stories matter: Using participatory new media storytelling and evaluation to serve marginalised and regional communities

In this chapter, we draw on our experiences facilitating community storytelling workshops in regional Queensland in partnership with the Queensland branch of Oral History Association of Australia (OHAA Qld) in order to develop a best practice model for promoting creative approaches to recording oral narratives using digital tools, informed by creative writing practice and embedded evaluation (Klaebe 2012 & 2013). These experiences offer an insight into how creative approaches to training can facilitate the sharing and preservation of stories in regional communities.

The way forward : report on the preparatory phase of prefabricated building design and construction methodology: identifying critical issues and recommending the way forward

This is an initial report of the PolyU SD part of the team to study Pre-fabricated Building Design and Construction Methodology and marks the completion of Phase 1. It follows our first notes prepared for the meeting on 2 February that identified some critical issues including future lifestyles, life expectancy of buildings, sustainability, size, flexibility and planning considerations. It is also an expansion of our presentation in Dongguan on 23 February. It is not a comprehensive survey of existing approaches or possible ways forward, but it has homed in on certain specific issues and does give specific examples to make the suggestions concrete. It is recommended that more comprehensive research be done to establish previous work and experience internationally. It is also recommended that more research be done on lifestyles as a preliminary to developing at least three concepts for evaluation before proceeding to the detailed design of one concept for full prototyping and market testing. The goal at this point is not to define a single direction but to suggest several future trajectories for further consideration. By the same token, this report is not intended as an exhaustive description of the considerable base of knowledge and ideas brought by the PolyU team to this exciting task. Before taking on an issue of this magnitude and importance in the definition of Hong Kong's future, one must carry out a thoughtful analysis of the issues at hand and an informed definition of paradigms, directions, goals and methods whereby our energies can be best used in the next steps. This report is the result of this analysis

Digital storytelling as participatory public history in Australia

This chapter discusses digital storytelling as a methodology for participatory public history through a detailed reflection on an applied research project that integrated both public history and digital storytelling in the context of a new master-planned urban development: the Kelvin Grove Urban Village Sharing Stories project.

Trust and strong ties in the selection and evaluation of business advisors by small firms

Principal Topic A small firm is unlikely to possess internally the full range of knowledge and skills that it requires or could benefit from for the development of its business. The ability to acquire suitable external expertise - defined as knowledge or competence that is rare in the firm and acquired from the outside - when needed thus becomes a competitive factor in itself. Access to external expertise enables the firm to focus on its core competencies and removes the necessity to internalize every skill and competence. However, research on how small firms access external expertise is still scarce. The present study contributes to this under-developed discussion by analysing the role of trust and strong ties in the small firm's selection and evaluation of sources of external expertise (henceforth referred to as the 'business advisor' or 'advisor'). Granovetter (1973, 1361) defines the strength of a network tie as 'a (probably linear) combination of the amount of time, the emotional intensity, the intimacy (mutual confiding) and the reciprocal services which characterize the tie'. Strong ties in the context of the present investigation refer to sources of external expertise who are well known to the owner-manager, and who may be either informal (e.g., family, friends) or professional advisors (e.g., consultants, enterprise support officers, accountants or solicitors). Previous research has suggested that strong and weak ties have different fortes and the choice of business advisors could thus be critical to business performance) While previous research results suggest that small businesses favour previously well known business advisors, prior studies have also pointed out that an excessive reliance on a network of well known actors might hamper business development, as the range of expertise available through strong ties is limited. But are owner-managers of small businesses aware of this limitation and does it matter to them? Or does working with a well-known advisor compensate for it? Hence, our research model first examines the impact of the strength of tie on the business advisor's perceived performance. Next, we ask what encourages a small business owner-manager to seek advice from a strong tie. A recent exploratory study by Welter and Kautonen (2005) drew attention to the central role of trust in this context. However, while their study found support for the general proposition that trust plays an important role in the choice of advisors, how trust and its different dimensions actually affect this choice remained ambiguous. The present paper develops this discussion by considering the impact of the different dimensions of perceived trustworthiness, defined as benevolence, integrity and ability, on the strength of tie. Further, we suggest that the dimensions of perceived trustworthiness relevant in the choice of a strong tie vary between professional and informal advisors. Methodology/Key Propositions Our propositions are examined empirically based on survey data comprising 153 Finnish small businesses. The data are analysed utilizing the partial least squares (PLS) approach to structural equation modelling with SmartPLS 2.0. Being non-parametric, the PLS algorithm is particularly well-suited to analysing small datasets with non-normally distributed variables. Results and Implications The path model shows that the stronger the tie, the more positively the advisor's performance is perceived. Hypothesis 1, that strong ties will be associated with higher perceptions of performance is clearly supported. Benevolence is clearly the most significant predictor of the choice of a strong tie for external expertise. While ability also reaches a moderate level of statistical significance, integrity does not have a statistically significant impact on the choice of a strong tie. Hence, we found support for two out of three independent variables included in Hypothesis 2. Path coefficients differed between the professional and informal advisor subsamples. The results of the exploratory group comparison show that Hypothesis 3a regarding ability being associated with strong ties more pronouncedly when choosing a professional advisor was not supported. Hypothesis 3b arguing that benevolence is more strongly associated with strong ties in the context of choosing an informal advisor received some support because the path coefficient in the informal advisor subsample was much larger than in the professional advisor subsample. Hypothesis 3c postulating that integrity would be more strongly associated with strong ties in the choice of a professional advisor was supported. Integrity is the most important dimension of trustworthiness in this context. However, integrity is of no concern, or even negative, when using strong ties to choose an informal advisor. The findings of this study have practical relevance to the enterprise support community. First of all, given that the strength of tie has a significant positive impact on the advisor's perceived performance, this implies that small business owners appreciate working with advisors in long-term relationships. Therefore, advisors are well advised to invest into relationship building and maintenance in their work with small firms. Secondly, the results show that, especially in the context of professional advisors, the advisor's perceived integrity and benevolence weigh more than ability. This again emphasizes the need to invest time and effort into building a personal relationship with the owner-manager, rather than merely maintaining a professional image and credentials. Finally, this study demonstrates that the dimensions of perceived trustworthiness are orthogonal with different effects on the strength of tie and ultimately perceived performance. This means that entrepreneurs and advisors should consider the specific dimensions of ability, benevolence and integrity, rather than rely on general perceptions of trustworthiness in their advice relationships.

Concurrent multi-scale modeling of civil infrastructures for analyses on structural deterioration—Part I : Modeling methodology and strategy

This paper aims to develop the methodology and strategy for concurrent finite element modeling of civil infrastructures at the different scale levels for the purposes of analyses of structural deteriorating. The modeling strategy and method were investigated to develop the concurrent multi-scale model of structural behavior (CMSM-of-SB) in which the global structural behavior and nonlinear damage features of local details in a large complicated structure could be concurrently analyzed in order to meet the needs of structural-state evaluation as well as structural deteriorating. In the proposed method, the “large-scale” modeling is adopted for the global structure with linear responses between stress and strain and the “small-scale” modeling is available for nonlinear damage analyses of the local welded details. A longitudinal truss in steel bridge decks was selected as a case to study how a CMSM-of-SB was developed. The reduced-scale specimen of the longitudinal truss was studied in the laboratory to measure its dynamic and static behavior in global truss and local welded details, while the multi-scale models using constraint equations and substructuring were developed for numerical simulation. The comparison of dynamic and static response between the calculated results by different models indicated that the proposed multi-scale model was found to be the most efficient and accurate. The verification of the model with results from the tested truss under the specific loading showed that, responses at the material scale in the vicinity of local details as well as structural global behaviors could be obtained and fit well with the measured results. The proposed concurrent multi-scale modeling strategy and implementation procedures were applied to Runyang cable-stayed bridge (RYCB) and the CMSM-of-SB of the bridge deck system was accordingly constructed as a practical application.

Characterising creativity : a holistic methodology for the assessment of creative works

This paper reports on the early stages of a design experiment in educational assessment that challenges the dichotomous legacy evident in many assessment activities. Combining social networking technologies with the sociology of education the paper proposes that assessment activities are best understood as a negotiable field of exchange. In this design experiment students, peers and experts engage in explicit, "front-end" assessment (Wyatt-Smith, 2008) to translate holistic judgments into institutional, and potentiality economic capital without adhering to long lists of pre-set criteria. This approach invites participants to use social networking technologies to judge creative works using scatter graphs, keywords and tag clouds. In doing so assessors will refine their evaluative expertise and negotiate the characteristics of creative works from which criteria will emerge (Sadler, 2008). The real-time advantages of web-based technologies will aggregate, externalise and democratise this transparent method of assessment for most, if not all, creative works that can be represented in a digital format.